Posted June 24, 2009 at 7:56 pm by Akshay
Stock Exchanges around the world currently publish datafeeds in CSV format. This is quite portable and easy to implement, however it cannot be directly used in your web applications without server side code. Also these official feeds are generally very expensive and affordable only by enterprise level developers. Thats where Stock Market Data (SMD) API comes in place. This simple HTTP webservice presents a snapshot of the latest stock market data in various Web 2.0 formats such as JSON, RSS, ATOM, MDDL (Market Data Definition Language – XML for Market Data) or a good old CSV. Currently SMDAPI supports NASDAQ Stock Exchange, New York Stock Exchange (NYSE), Bombay Stock Exchange (BSE) and National Stock Exchange, India (NSE) but we are actively working on supporting others soon.
Please note that SMD API is currently released as Beta and is quite unstable in nature. I am still experimenting with everything from the domain name of this service (smdapi.co.cc which is a temperory one) to the API structure itself. I am also very much open to ideas which I can incorporate in this project before it officially released.
Posted June 8, 2009 at 6:05 pm by Akshay
This is probably a major milestone in the lifecycle of WP Web Scraper WordPress plugin. Technically speaking, the plugin gets in own ‘module architecture’ to incorporate unlimited extensions without touching the core codebase. Speaking non-technically, this opens WP Web Scraper to a non-techie WordPress user. To startoff, this mod extends the plugin with a specific shortcode to get stock market data from NSE and NASDAQ (to start off with, more exchanges soon to come). The data is scraped with a cache interval of a minute (which can be further increased as per your requirement) and includes data types such as Open, High, Low, Last Price, Previous Close, Change, Change Percentage and Volume information for all active symbols on these exchanges.
The plugin API will provides a simple shortcode. For example – [wpws_market_data market="nse" symbol="acc" datatype="last"] or [wpws_market_data market="nasdaq" symbol="csco" datatype="open"]. NSE data is currently scraped from nseindia.com and NASDAQ data is scraped from reuters.com. The immediate plan is to implement all major stock markets in this API. Later, I plan to extend this modular architecture to other categories of scraps such as Weather, Sports scores etc too.
Posted May 27, 2009 at 12:20 pm by Akshay
My latest WordPress plugin for web scraping – WP Web Scrapper was a grand launch. It recorded more than 200 downloads in the first two days itself! Thanks for all the appreciation and comments. This post is mainly to list down my plan to extend WP Web Scrapper into a standard scraping framework. Apart from being a flexible framework, I also plan to introduce some pre-built modules to make specific and highly desired scraping tasks easy. First such module will be a stock market data grabber. This module will extend the plugin to get stock market data from various big exchange websites easily (planning to support NSE, BSE and NASDAQ to start off with). The data will be almost realtime (delay ranging between 1 to 10 mins) and will include Open, High, Low, Last Price, Previous Close, Change, Change Percentage and Volume information for all active symbols on these exchanges.
The plugin API will provide a shortcode something like this – [wpws mod="nse" symbol="acc" datatype="last"] should output the latest price for ACC listed at NSE. The aim is to make it an extendable module framework and hence I am taking time to code it well. Apart from this features, I am also planning to improve the core scrapper with functionalities like a regex powered cleanup function to remove all unwanted text strings from the scrap and also a more flexible algorithm to query html tables returned by the scrap.
Posted May 19, 2009 at 6:49 pm by Akshay
This is in continuation to my concept note on BSE and NSE syndication feeds. Here are some more sketchy details about the planned service.
Data source: Currently I am considering to scrap the stock exchange website pages using cURL and then parsing the DOM structure of the raw source. Once the service grows big enough, a subscription to an official data feed can be considered.
Output formats: I realized that RSS was a bad idea after all, however I discovered a very evolved XML specification for market data representation called Market Data Definition Language (MDDL). Apart from MDDL the other output formats could be a simpler XML (home grown format); good old csv (light, easy and time tested) and JSON (for direct use in client side apps). As JSONP is natively cross domain, it can be easily integrated without any server side technology. This means that users of this service can start their own ‘Get Quote’ apps on their website irrespective of the server or the host they use… just plain javascript code.
Lets get down to an example here. This is the current get quote output from NSE (exchange) for ACC (stock). An example output in MDDL would look like this sample mddl-xml. This as such is the plan and I am eagerly awaiting some feedback on this one before I start off with the actual code.
Posted May 15, 2009 at 12:57 am by Akshay
Consider this. Our stock market sites gear themselves as Web 2.0 services and start publishing data in the form of RSS, XML or JSON feeds. Imagine receiving corporate announcements in your RSS aggregator or even better… track your portfolio thru your aggregator. Further, picture what the developer community can do using this. Stock feeds can be parsed using widget frameworks like Google Gadgets, OpenSocial or plugin architectures to be dispayed on sites, blogs or social networking profile pages. Basically the possibilities are limitless.
But the reality is that our stock markets won’t do it. Not at least so soon. To fill in this gap I am planning to start a service which will parse BSE or NSE pages and convert them into RSS, XML or JSON feeds. It will be a simple HTTP service which will answer your request in the desired format. I will start work as soon I find a some time off my current projects. Till then, give it a thought and let me know what do you think of this.
5 Comments 
Leave a comment »
RSS
Social bookmarks
Hi, My name is Akshay Raje. I am a self taught freelance web designer and developer. I love travel and am a great movie buff too. This blog is about my projects and experiments with web technologies. 
