Stock Market Data API (Beta release)

Stock Exchanges around the world currently publish datafeeds in CSV format. This is quite portable and easy to implement, however it cannot be directly used in your web applications without server side code. Also these official feeds are generally very expensive and affordable only by enterprise level developers. Thats where Stock Market Data (SMD) API comes in place. This simple HTTP webservice presents a snapshot of the latest stock market data in various Web 2.0 formats such as JSON, RSS, ATOM, MDDL (Market Data Definition Language – XML for Market Data) or a good old CSV. Currently SMDAPI supports NASDAQ Stock Exchange, New York Stock Exchange (NYSE), Bombay Stock Exchange (BSE) and National Stock Exchange, India (NSE) but we are actively working on supporting others soon.

Please note that SMD API is currently released as Beta and is quite unstable in nature. I am still experimenting with everything from the domain name of this service (smdapi.co.cc which is a temperory one) to the API structure itself. I am also very much open to ideas which I can incorporate in this project before it officially released.

Update: The API is currently no more supported in interest of this method I lately discovered. Please contact me at akshay[dot]raje[at]gmail.com if you need any help / consulting in this regard.

Collaboration – ready to take over Web 2.0

At first there was good old email with petty 5 Mb inboxes. Then we moved to crowded chat rooms on Yahoo. All that seemed as stone age, when came a host of interactive web applications like Blogs and Social Networking applications. Initially Orkut, then Facebook and then micro blogging thru Twitter. Have you ever wondered what could be the next big Web 2.0 trend?

The answer is ‘Collaboration’. Yes, collaboration not only in its own sense, but also a collaboration of all the trends and technologies mentioned above. I know the word ‘collaboration’ sounds too ‘enterprise’ and not at all ‘social’, but as always web has once again taken us by surprise here. What I am referring is not too far fetched. Big boys like Google and Yahoo have already stepped in this domain and grand releases have already been done. Yes, check out Google Wave and Zimbra (by Yahoo, but not much talked about yet) to understand what am pointing at. Google Wave is a new tool for real-time communication and collaboration on the web, coming later this year. An official 90 minute video demo is already out for a sneak peak into this new wonderland. Don’t have 90 minutes? Have a look at my detailed post on Devil’s Workshop with 30-60 second clips highlighting the best parts of Google Wave.

WP Web Scraper – A WordPress Stock Market plugin

This is probably a major milestone in the lifecycle of WP Web Scraper WordPress plugin. Technically speaking, the plugin gets in own ‘module architecture’ to incorporate unlimited extensions without touching the core codebase. Speaking non-technically, this opens WP Web Scraper to a non-techie WordPress user. To startoff, this mod extends the plugin with a specific shortcode to get stock market data from NSE and NASDAQ (to start off with, more exchanges soon to come). The data is scraped with a cache interval of a minute (which can be further increased as per your requirement) and includes data types such as Open, High, Low, Last Price, Previous Close, Change, Change Percentage and Volume information for all active symbols on these exchanges.

The plugin API will provides a simple shortcode. For example – [wpws_market_data market="nse" symbol="acc" datatype="last"] or [wpws_market_data market="nasdaq" symbol="csco" datatype="open"]. NSE data is currently scraped from nseindia.com and NASDAQ data is scraped from reuters.com. The immediate plan is to implement all major stock markets in this API. Later, I plan to extend this modular architecture to other categories of scraps such as Weather, Sports scores etc too.

Plan of action for WP Web Scraper

My latest WordPress plugin for web scraping – WP Web Scrapper was a grand launch. It recorded more than 200 downloads in the first two days itself! Thanks for all the appreciation and comments. This post is mainly to list down my plan to extend WP Web Scrapper into a standard scraping framework. Apart from being a flexible framework, I also plan to introduce some pre-built modules to make specific and highly desired scraping tasks easy. First such module will be a stock market data grabber. This module will extend the plugin to get stock market data from various big exchange websites easily (planning to support NSE, BSE and NASDAQ to start off with). The data will be almost realtime (delay ranging between 1 to 10 mins) and will include Open, High, Low, Last Price, Previous Close, Change, Change Percentage and Volume information for all active symbols on these exchanges.

The plugin API will provide a shortcode something like this – [wpws mod="nse" symbol="acc" datatype="last"] should output the latest price for ACC listed at NSE. The aim is to make it an extendable module framework and hence I am taking time to code it well. Apart from this features, I am also planning to improve the core scrapper with functionalities like a regex powered cleanup function to remove all unwanted text strings from the scrap and also a more flexible algorithm to query html tables returned by the scrap.

Bringing Web Scraping to WordPress!

Web scraping (or Web harvesting, Web data extraction) is a computer software technique of extracting information from websites. Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be formatted and displayed or stored and analyzed. Exemplary uses of Web scraping include online price comparison, weather data monitoring, market data tracking, Web content mashup and Web data integration.

Imagine what you can do with all this power in your WordPress blog! Pages and posts can display realtime content from other pages, letting you create a meshup of content. This all is now possible using my WP Web Scraper plugin. Its an easy to implement professional web scrapper for WordPress. This can be used to display real time data from any websites directly into your posts, pages or sidebar. Use this to include real time stock quotes, cricket or soccer scores or any other generic content. The scrapper is built using time tested libraries cURL for scrapping and phpQuery for parsing HTML. Please post all your suggestions and thoughts about this on the WP Web Scraper project page.

More on BSE / NSE feeds project

This is in continuation to my concept note on BSE and NSE syndication feeds. Here are some more sketchy details about the planned service.

Data source: Currently I am considering to scrap the stock exchange website pages using cURL and then parsing the DOM structure of the raw source. Once the service grows big enough, a subscription to an official data feed can be considered.

Output formats: I realized that RSS was a bad idea after all, however I discovered a very evolved XML specification for market data representation called Market Data Definition Language (MDDL). Apart from MDDL the other output formats could be a simpler XML (home grown format); good old csv (light, easy and time tested) and JSON (for direct use in client side apps). As JSONP is natively cross domain, it can be easily integrated without any server side technology. This means that users of this service can start their own ‘Get Quote’ apps on their website irrespective of the server or the host they use… just plain javascript code.

Lets get down to an example here. This is the current get quote output from NSE (exchange) for ACC (stock). An example output in MDDL would look like this sample mddl-xml. This as such is the plan and I am eagerly awaiting some feedback on this one before I start off with the actual code.

Flash Photo Gallery

I have now put up an exclusive page for one of my plugin projects – Flash Photo Gallery. This is a WordPress plugin which creates a Flash Photo Gallery like one provided in Adobe Photoshop CS2 Flash Web Photo Gallery templates. All plugin ‘ToDos’ and release information will be available on this page and any feedback is much appriaciated.

BSE and NSE syndication feeds

Consider this. Our stock market sites gear themselves as Web 2.0 services and start publishing data in the form of RSS, XML or JSON feeds. Imagine receiving corporate announcements in your RSS aggregator or even better… track your portfolio thru your aggregator. Further, picture what the developer community can do using this. Stock feeds can be parsed using widget frameworks like Google Gadgets, OpenSocial or plugin architectures to be dispayed on sites, blogs or social networking profile pages. Basically the possibilities are limitless.

But the reality is that our stock markets won’t do it. Not at least so soon. To fill in this gap I am planning to start a service which will parse BSE or NSE pages and convert them into RSS, XML or JSON feeds. It will be a simple HTTP service which will answer your request in the desired format. I will start work as soon I find a some time off my current projects. Till then, give it a thought and let me know what do you think of this.

Hello world!

So. I’m officially here. In the blogosphere. Finally.

I struggled with what would be my first official post. It should be prolific after all this time, right? Grand in nature. Broad in scope. So, I struggled. And then I realized it’s just like walking into a room full of people you’ve never met – I’ll introduce myself to you. It seems like the most polite thing to do, right?

First an introduction for the unfamiliar. Web design and development (Web.D) is what I have been doing for a while now. Am sure most of you are not aware of this layer of me, but what the heck… some of you might not even be knowing me. So lets start from the scratch. I am Akshay Raje, a self thought freelance web designer and developer. I love travel and am a great movie buff too. Apart from the above, I try to spend the rest available time with my wife and also try to squeeze out of whatever time’s left for the Gym.

That’s it. That’s the gist of who I am, but this blog isn’t about me. It’s about what I love to do. And I love to code for creative satisfaction. Mankind by nature has always been seeking ways and means to channel creativity. These channels range from Fine Arts to Applied Sciences. For me, coding is the simplest available way to draw creative satisfaction from tangible achievements. Web technologies has obviously been a natural choice in this pursuit due to the high amount of exposure, visibility and thereby tangibility.

I have worked on a wide range of projects including cross browser HTML and CSS implementation of web design, Flash web sites, WordPress powered blog setups with custom theme development, WordPress as a Content Management System (CMS) for websites, PHP-MySQL powered web applications, AJAX powered web applications, Web-scrapping and Feed parsing applications. I have also done a couple of WordPress Plugins and some Google Gadgets too. Read more of this on the Projects page.

This blog is all about my projects and experiments with web technologies. At times I may digress but that won’t be a dominating part of this blog. I am keen to hear from you, anytime and all the time. Constructive criticism is welcome, slander is not. Basically, please don’t say something here that you wouldn’t say to me in person. So, there you have it. My little introduction to the blogosphere. If you have questions or comments, please ask. I’m all ears.