OSU Web Services

A maintenance release was pushed to production for Search, version 0.3.1.

For those who haven’t noticed the capabilities of search, look at the options, if a word is typed in, there may be some suggestions offered.  At the bottom is related searches, if you misspell a word, you may see a “Did you mean…”.  Also, domain search has been added to the Advanced Search.  Did you notice these changes?  We implemented all this in a front end, meaning we look at the features Google has, create an interface to the appliance, and then place things where we need to place it.  The front end also serves a dual purpose, if for future reasons, the backend, aka the Google Appliance is replaced, we simply rewrite the front end to work with the new back end.  More importantly, the front end allows us to do other certain aspects, like implement our feedback module.

Now with a front end, it also means we may have minor bugs, which is why we release the maintenance version for minor bugs.  It may simply be things like formatting, or some case we did not handle that the appliance handles.

We’ll be continuing to look at enhancing search, and integrate other aspects with it as we move ahead.  There has been minimal feedback to date, and without feedback, we cannot know how we can improve it.  So if you have a comment let us know.

Thanks!  Your Central Web Services Group

As part of the next search release, we now have a quick feedback module.  If you are looking for a particular search which you know is on a specific site, for example, on forestry.oregonstate.edu, and you do not find that, what do you do?  Well, let us know.

It is entirely possible that sites are not indexed into our search engine, so just send us a quick feedback with the url.  If you want to be contacted, provide us with contact information as well, or instead use our full help form.

If you want to let us know anything else about the layout or design, then just tell us more.  Enter the information and just click Send Feedback, and your feedback will come to us.  We try to be mind readers as best as possible, but there are particulars which we may not be able to pick up on.  If there is something we can do about your feedback, if it helps many students, faculty, and staff, we do want to look at how we could accomplish it.

Thank You, Your Central Web Services Team

You’ve probably come here maybe because you clicked on the Keyword link from our search page, or you came across our blog, or from blogs.o.e.  But however you arrived here is secondary to the information that you want.

What is a Keyword, how do I get one?  The first part is easy, a Keyword is a prominent link that is top of results based on, well, keywords.  Google calls it Keymatch, but we are keeping our prior terminology of Keyword.  So for example, a keyword could be “academic calendar” which when you type in search, displays in a shaded area above the result set, that when you click the link takes you to the catalog for the academic calendar.

So how do you get one?  In our previous search engine, we would have to enter keywords into a database manually, and there was no policy on establishing keywords, and as the keywords grew and grew, the maintenance in ensuring links were fresh was too much of an overhead.

With the new Google Search Appliance, we are operating in a different mode, we have data to look at, and with good search engine optimization for your pages, organic results should be improved.

It was previously necessary for keywords for many users, but with a better organic result set, we can minimize the number of keywords we have to maintain.  With the ability now to see what are the top queries that both get and don’t get results, we can make some intelligent determination on what should be keywords.  For others, we do recommend that you optimize your page for search engines, and there is information about it on Google’s site.  The basics on it though if you don’t want to read all about it is, one, relevant content, and two, other sites to link to your site.

We’ll be looking at other approaches in the future to build upon the need for additional promotion, but for now, a data-driven approach is what we will be looking at for a fresh approach to searching by keywords.  So for now, we will not be taking user requests for keywords.  Stay tuned to this blog for changes.  In the future we hope to make the search reports accessible via a web interface that any user can visit.  If you have feedback, please contact us, or leave a comment here.

With the release of the Advanced Search function, we have released Search as production.  Hmm, what does that mean?  Well, now if you go to the original search site, at search.oregonstate.edu, it is the new Google Search Appliance search.  So does that mean that’s it you ask?  Well, no.  There’s still more features coming, like Narrow Your Search and Keywords, which we’ll discuss more about in the near future.  For now, look at the advanced search.  If you really are looking to search, and you don’t find what you are looking for, don’t give up.  Try the advanced search feature, add more words, exclude words, pick a specific file format.  There are several file formats you can look to find.

Now if you don’t find what you are looking for, it could be that the site you are looking to search might be new and not crawled yet, or it might not be hosted with us, so we are not aware it needs to be crawled.  In this case, all you have to do is just let us know what the site is, and we’ll look to crawl.  Read our other post on exceptions to get additional information as well on what we don’t crawl.

So, we’ve been talking about search, and people no doubt wonder if their site will be found with OSU’s Google Search.

Most cases, the answer is yes, but in some cases the answer is no.  For the no’s there are reasons why and is what I want to talk about in this post.

The Exceptions

Why are there exceptions?  There are exceptions for a few reasons.  First is the license limit on our Appliance, which is currently one million documents.  549,998 is our current document amounts and we are still indexing sites as we are made aware.  So if a site has a large number of documents, for example a site that has an individual page show up for a dictionary, where each entry of the dictionary is considered a document, then that will eat up the million document limit fairly quickly.  Relating to the previous example, some exceptions are because of the applications users or departments use.  For example, currently a Joomla CMS results in a large number of documents returned because of the way the application works.   Second, if there are sites that are not maintained, which get hacked or spammed, we don’t want to index sites that have spam inserted into it which may likely show up in the search descriptions.  Third, if crawling the site results in an endless loop, where documents in the site refer to itself, so the crawler basically gets stuck, don’t crawl those.  Fourth, if a site returns a large number of errors, then there is something wrong with the site and that is consuming the Appliance resources, such as CPU and memory.  Fifth encompasses all these aspects, which is the administration overhead.  With all the other functions CWS supports, if a particular search aspect would result in significant administration overhead, we would need to make the best decision to minimize that overhead.

So what are our current exceptions?

1.  ONID home directories are not searched.  Why?  Mostly because some users do not maintain their sites, and the sites result in spam entries, and across twenty thousand or more, it’s too much of an overhead to manage.  A policy decision was made for this.
2.  http://ecampus.oregonstate.edu/ask-ecampus/knowledge-base/  Why? This site returned over 250 thousand documents.
3.  http://www.cof.orst.edu/org/iawa/  Why?  This site returned over 160 thousand documents.
4.  http://oregonstate.edu/tac/index.php?option=  Why?  This site returned over 600 thousand documents (due to the way the application handles pages)
5.  Group sites at http://oregonstate.edu/groups/ Why?  This is for the same reason as #1.  As part of the move to people.oregonstate.edu for group sites, we will be reevaluating this.
6.  http://oregonstate.edu/webprojects/wiki Why?  has 2 million errors
7.  http://www.familybusinessonline.org/index.php? Why?  This site returned over 400 thousand documents (due to the way the application handles pages).
8.  http://oregonstate.edu/cla/anthropology/gallery/kingston/main.php? Why?  This site was caught in a loop.
9.  http://bioe.oregonstate.edu/reservations/ Why?  This site was caught in a loop.
10.  http://oregonstate.edu/aepcore/index? Why?  This site was caught in a loop.
11.  http://hort.oregonstate.edu/event/  Why?  This site was caught in a loop.
12.  http://recycle.oregonstate.edu/EarthDay/eventCalendar.cfm?  Why?  This site was caught in a loop.
13.  http://extension.oregonstate.edu/clackamas/announcement/  Why?  This site was caught in a loop.
14.  http://physics.oregonstate.edu/event/  Why?  Events list returning excessive results.
15.  regexp:http://www\\.osualum\\.com/?.*cid=[0-9]+.*?  This is a regular expression statement that if it has the url form specified then it is not being crawled.  Why?  This site was caught in a loop.
16.  http://oregonstate.edu/sli/aggregator/announcement/  Why?  This site was caught in a loop.
17.  regexp:http://oregonstate\\.edu/womenscenter/library.*browse=*  This is a regular expression statement that if the url has the aspects specified within it, then it is not being crawled.  Why?  This site was caught in a loop.

If your site is on this list, and you want to discuss this, then contact us.  We do want to reevaluate sites periodically

We also do not index every type of file extension.  Image files, media files, archive or binary files are not crawled.  There would just be way too many that would exceed our license.

So those are the exceptions are reasons why.  We don’t necessarily expect everyone to be happy or agree with the exceptions made, however, we have to make the best decisions to support OSU as a whole and keep in mind the limitations of our search engine.  However, stating that, we do want to periodically review our decisions, and also determine if alternative solutions can be implemented.  So if there is a concern, then please contact us.

So Search Beta was released in conjunction with the new top hat design for OSU (another change as part of future upcoming changes).  A great effort between Central Web Services (otherwise known as CWS) and Web Communications.  The same collaborative group that introduced OSU Mobile.  Don’t know about OSU Mobile?  Well for that, visit m.oregonstate.edu (iPhone, Palm Pre, Android and some Blackberry), and I’m sure we’ll be talking about that in other OSU CWS blog posts, so stay tuned.

So what is Search Beta?  It’s just that, it’s really Beta.  We are transitioning information, crawls, features from the Google Search Appliance to the User Interface for search.  It’s not perfect, not everything will be found right now.

So you might be wondering about how that affects search on your site pages, which uses the central code provided by CWS.  Because we have a front end to search, we are able to make it as transparent as possible to site owners.  The goal is sites shouldn’t need to be modified, if they use the search module integration CWS has provided and made available previously.  Integration with Drupal sites will be upcoming, so if your site is not showing results because it has not been indexed, do not worry, we’ll be rolling the Drupal change in soon.  After that there are a couple things that need to happen.  First if you are running what we call a virtual host, like hmsc.oregonstate.edu or in a path in oregonstate.edu/, the Google search has to find your site possibly linked from other sites and index the site.  This is the engine part of the appliance, and Google does a fairly good job with this.  The process could take anywhere from a few hours to a few days, depending on the algorithms Google uses to find new pages.  Anything in the oregonstate.edu/ area is continuously crawled.  There are exceptions (and reasons for exceptions), which we’ll be noting in the days to come and which we’ll talk more about in another post.   Second, if your site is not found after a reasonable amount of time, then we can look at explicitly crawling your site.  This is more common with virtual hosts.  If that is the case just contact us using our online contact form, but first read the next post about exceptions to sites being crawled.

We’ll also be looking to get some input from users.  You can comment here, or you can comment on the Web Communications blog, where there will also be information about the new home page that will be introduced this year.  In addition to commenting, there will be some focus groups, which is another avenue to provide feedback.  The focus groups will look at search among other things.

So when it comes to the OUS Search, we say Search Me.