For a while I’ve been wanting to create a specialized search engine for a project I have been working on. Given the limited amount of time I have, I clearly had to piggyback on current technologies. Search engines today do an awesome job of indexing the web to find what you need when you need it. I wanted a specialized search engine to limit the results to relevant web content given a specific subject.
Here were my requirements going into this:
- Limit the search to the sites I want. I give them a list and it only searches those sites. The list would need to be updated dynamically or via an API.
- Allows me to modify the search results in any order I want or mixing it with other data.
- Allows me to format the search results how I please.
- I want the results in a multiple formats to allow for flexibility when and where the results are displayed.
- I would love to try to monetize the search engine however I want.
Searching for what I needed led me to discover many cool technologies.
I first looked at possibly the most obvious: Google Custom Search Engine. Google CSE is a cool product but it didn’t suit my needs at all. The terms of service don’t allow you to modify the results. I also looked at Google AJAX Search API. Again, doesn’t fit the requirements. I may find a need for the AJAX Search API in the near future though.
Other technologies deserved some attention as well. Products such as CouchDB and Lucene required too much so I didn’t spend much time looking into these.
I had looked at Yahoo BOSS (build your own search service). Initially, it didn’t look like it would suit my needs. After revisiting BOSS, looking at code samples and the API, it was exactly what I was looking for. Easy, fast, reliable. Fits exactly my requirements. Awesome.
Since the site I am building was based on WordPress (like all my sites), I decided to put my work it into a WordPress plugin to allow a WordPress site to easily add a real search engine to the site just by defining some simple options. The first version will be very limited in what can be done but I have a lot of features that I will be adding to it very soon.
I just need to get it ready to release. Soon I hope.