RE: full text search

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

RE: full text search

peter-8
Damien, the changes I am suggesting are for MySQL. You are right it has a
concept of Full Text Search (which works fine); but as I mention the IN
BOOLEAN MODE part of the MATCH AGAINST statement is missing; plus we do a
Simple Search "over top" of FT search which, in effect, masks the FT search.

I am sorry I do not know much about Postgre or other db languages or how
tiki adjusts for different languages (I would guess just a CASE statement
for any code which might need to be specific to the DB)

Peter Lindstrom
All Analog - Technical Design Consulting
Ottawa, ON
Voice:  613-612-1419
Email:  [hidden email]
Web:    www.allanalog.com  
 


::-----Original Message-----
::From: [hidden email] [mailto:tikiwiki-devel-
::[hidden email]] On Behalf Of tikiwiki-devel-
::[hidden email]
::Sent: Wednesday, May 18, 2005 1:47 AM
::To: [hidden email]
::Subject: Tikiwiki-devel digest, Vol 1 #1256 - 1 msg
::
::Send Tikiwiki-devel mailing list submissions to
:: [hidden email]
::
::To subscribe or unsubscribe via the World Wide Web, visit
:: https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
::or, via email, send a message with subject or body 'help' to
:: [hidden email]
::
::You can reach the person managing the list at
:: [hidden email]
::
::When replying, please edit your Subject line so it is more specific
::than "Re: Contents of Tikiwiki-devel digest..."
::
::
::Today's Topics:
::
::   1. Re: Consensus on Full Text Search change (Roberto Leibman)
::
::--__--__--
::
::Message: 1
::Date: Tue, 17 May 2005 18:57:11 -0700
::From: Roberto Leibman <[hidden email]>
::To: [hidden email]
::Subject: Re: [Tikiwiki-devel] Consensus on Full Text Search change
::Reply-To: [hidden email]
::
::Jason Diceman wrote:
::
::> Any improvement to search engine results is really appreciated.
::>
::> AND should definitely be the default.
::>
::I haven't looked at this in any detail but... what about using full text
::searches in mysql? I remember postgress has something similar, have no
::idea about others.
::
::> Cheers
::>
::> - jd- - -
::>
::> ------------------------------------------------------------------------
::>
::> *From:* [hidden email]
::> [mailto:[hidden email]] *On Behalf Of *peter
::> *Sent:* May 17, 2005 5:46 PM
::> *To:* [hidden email]
::> *Subject:* [Tikiwiki-devel] Consensus on Full Text Search change
::>
::> please see my post to 1.9 bug tracker at:
::>
::> http://dev.tikiwiki.org/tiki-
::view_tracker_item.php?itemId=166&show=view&status=op&trackerId=5&sort_mode
::=f_41_desc
::> <http://dev.tikiwiki.org/tiki-
::view_tracker_item.php?itemId=166&show=view&status=op&trackerId=5&sort_mode
::=f_41_desc>
::>
::> basically I am suggesting fixing the Full Text Search routine.
::>
::> briefly (if I can):
::>
::> - FT search should do AND function
::>
::> - at first it seems to be doing OR (which is silly concept for a
::> search routine!!)
::>
::> - after looking at code I can see that it Is doing AND; but if no
::> results are found it does an OR (simple search) everywhere
::>
::> ok, maybe just me; but that seems to defeat the purpose of doing AND.
::>
::> perhaps this wouldn’t be so bad; but the simple search then passes the
::> + signed search terms to simple search routine which chokes on the +
::> signs.
::>
::> so, I think, with FT Search enabled, current code does:
::>
::> red ball - red OR ball
::>
::> +red +ball - query error
::>
::> my fix (i.e. removing code to do simple search if no results):
::>
::> red ball - red OR ball (this should likely alos be changed to default
::> to +red +ball and do an AND
::>
::> +red +ball - red AND ball
::>
::> One last comment – there is also a bug in the code to do the FT
::> Search; as mentioned in my comment on the bug tracker item listed above.
::>
::> I guess if no one objects, and I figure out how to submit code, will
::> submit my changes by the end of the week… AND THEN, when TW.org gets
::> the updated code I can finally do a search on the TW forums… Yeaaaa!!!
::>
::> *Peter** Lindstrom*
::>
::> All Analog - Technical Design Consulting
::>
::> Ottawa, ON
::>
::> Voice: 613-612-1419
::>
::> Email: [hidden email]
::>
::> Web: www.allanalog.com <http://www.allanalog.com/> **
::>
::
::
::
::
::--__--__--
::
::_______________________________________________
::Tikiwiki-devel mailing list
::[hidden email]
::https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
::
::
::End of Tikiwiki-devel Digest



-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
<a href="http://ads.osdn.com/?ad_idt12&alloc_id344&op=click">http://ads.osdn.com/?ad_idt12&alloc_id344&op=click
_______________________________________________
Tikiwiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

re: full text search

peter-8

first 1 quick question:

 

-          why when I send an email to this list does it say moderator needs to approve because I am not a member; as far as I know I am a member

 

 

so, commenting to flo’s comment about TW does have a FT search. Yes, I know that it does; it just doesn‘t work.

 

one more time; I think 2 mistakes:

 

searchlib->_find() has the following code:

 

if ($fulltext) {
                  $qwords = $this->db->quote($words);

      // p.lindstrom added IN BOOLEAN MODE to make MATCH AGAINST work properly
                  $sqlft = 'MATCH(' . join(',', $h['search']). ') AGAINST (' . $qwords . ' IN BOOLEAN MODE)';
                  $sqlWhere .= ' AND ' . $sqlft ;
                  $sqlFields .= ', ' . $sqlft . ' AS relevance';
                  $orderby = 'relevance desc, ' . $orderby;
            }

 

I added the “IN BOOLEAN MODE” part which I believe is required for MATCH AGAINST to do FT search.

 

ALSO:

 

in the same function:

 

if ($fulltext && $words) // try a simple search
      return $this->_find($h, $words, $offset, $maxRecords, false);
else
      return array(
            'data' => array(),
            'cant' => 0
            );

 

which is, I think is stating if nothing found with FT search do a simple search (which I think needs to be removed)

 

OK – something I think I have failed to clarify:

 

-          the reason that the simple search addition is WRONG is because the search (FT or simple) is done on a “category” by “category” basis

-          what this means is that the search is run first on blogs, then articles, then galleries, etc

-          therefore the simple search, if no results, is done on a category when no results are found in that category

-          if the problem with this is not clear.. then I got some splainin’ to do…

 

I want to search on:

 

+javascript +modules

 

-          lets say one article exists that matches (i.e. BOTH WORDS EXIST) and it is in the forum

-          but there are 1000 matches for “modules” in blogs and 1400 matches for “javascript” in articles

 

because no “+javascript +modules” match is found in blogs it will then do a simple search which will return the 1000 matches in blogs, etc

 

therefore I can not see my single tree for the massive forest.

 

this of course is not mentioning the fact that as soon as I pass “+javascipt” through the simple search routine; the SQL query will choke on the + sign.

 

I hope I got this right; and I hope this is clearer now.

 

Peter Lindstrom

All Analog - Technical Design Consulting

Ottawa, ON

Voice:  613-612-1419

Email:  pta@...

Web:   www.allanalog.com 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: re: full text search

Marc Laporte


peter wrote:

> first 1 quick question:
>
>  
>
> -          why when I send an email to this list does it say moderator
> needs to approve because I am not a member; as far as I know I am a member
>

You are registered as tiki@....com
<../../options/tikiwiki-devel/tiki--at--allanalog.com> but sending
emails as pta@......com   I have added it to mailman so I won't have to
manually approve your messages anymore.

>  
>
>

Thank you Peter for your work & analysis on Tiki search.

I look foward to testing your commits.

Best regards,

--
M ;-)

//////////////////////////////////////////////////////////////////
/                                                                /
/ Marc Laporte       <|>                  http://marclaporte.com /
/ Avantech.net       <|>                     http://avantech.net /
/ Tiki CMS/Groupware <|> http://tikiwiki.org/UserPagemarclaporte /
/                                                                /
//////////////////////////////////////////////////////////////////



-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7412&alloc_id=16344&op=click
_______________________________________________
Tikiwiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

re: full text search

peter-8
In reply to this post by peter-8

I am trying to do commits for my changes to Full Text Search routine but CVS says I don’t have write access.

 

I have never done commits before; but I believe I am registered developer – should I not have write access to CVS repo?

 

I am using BRANCH-1-9 is this correct?

 

Peter Lindstrom

All Analog - Technical Design Consulting

Ottawa, ON

Voice:  613-612-1419

Email:  pta@...

Web:   www.allanalog.com 

 

 

Reply | Threaded
Open this post in threaded view
|

RE: re: full text search

Sylvie Greverend
Is your CVSROOT like this :ext:[hidden email]:/cvsroot/tikiwiki ?
If you have a anonymous acees (like CVSROOT= pserver:[hidden email]:/cvsroot/tikiwiki ) it will not work
 
For BRANCH-189 you need to be sure to what you are doing... we are supposed to do only bug fixes on this branch as 1.9.1 release is coming soon.
For new feature or dev with some risk, use HEAD
 
-----Original Message-----
From: [hidden email] [mailto:[hidden email]]On Behalf Of peter
Sent: Friday, May 20, 2005 12:03 PM
To: Tiki Dev List
Subject: [Tikiwiki-devel] re: full text search

I am trying to do commits for my changes to Full Text Search routine but CVS says I don’t have write access.

 

I have never done commits before; but I believe I am registered developer – should I not have write access to CVS repo?

 

I am using BRANCH-1-9 is this correct?

 

Peter Lindstrom

All Analog - Technical Design Consulting

Ottawa, ON

Voice:  613-612-1419

Email:  pta@...

Web:   www.allanalog.com 

 

 

Reply | Threaded
Open this post in threaded view
|

re: full text search

peter-8
In reply to this post by peter-8

Hi Sylvie,

 

My CVS root is:             :ext:[hidden email]:/cvsroot/tikiwiki

 

my branch is 1-9

 

the fix I am trying to make is for what I consider to be a bug that has existed in Full Text Search likely since it was first written.

 

Marc Laporte has suggested I should commit for 1.9.0 fixes. I posted to this mail list to see if anyone objected. Not much discussion on the topic; suspect my fix is not the best; but definitely better than what is there now.

 

I have fixed it on my site; so my users can now do a proper search; but would be nice if it was incorporated in official code cuz then I would be able to search on tw.org (as I guess everyone else would be as well)

 

Peter Lindstrom

All Analog - Technical Design Consulting

Ottawa, ON

Voice:  613-612-1419

Email:  pta@...

Web:   www.allanalog.com 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: re: full text search

toggg
Hi peter & all !
I use :
export CVS_RSH=ssh
export CVSROOT=":ext:[hidden email]:/cvsroot/tikiwiki"
cvs -q -z3 update -r BRANCH-1-9 tikiwiki
to update my local

and, having same symbols CVS_RSH and CVSROOT
cd in somewhat the directory
cvs -q -z3 commit -r BRANCH-1-9 myfile
to put the changes (omg) to the cvs

Hope that serves
à+
bertrand Gugger aka toggg

peter wrote:

> Hi Sylvie,
>
>  
>
> My CVS root is:            
> :ext:[hidden email]:/cvsroot/tikiwiki
>
>  
>
> my branch is 1-9
>
>  
>
> the fix I am trying to make is for what I consider to be a bug that
> has existed in Full Text Search likely since it was first written.
>
>  
>
> Marc Laporte has suggested I should commit for 1.9.0 fixes. I posted
> to this mail list to see if anyone objected. Not much discussion on
> the topic; suspect my fix is not the best; but definitely better than
> what is there now.
>
>  
>
> I have fixed it on my site; so my users can now do a proper search;
> but would be nice if it was incorporated in official code cuz then I
> would be able to search on tw.org (as I guess everyone else would be
> as well)
>
>  
>
> *Peter** Lindstrom*
>
> All Analog - Technical Design Consulting
>
> Ottawa, ON
>
> Voice:  613-612-1419
>
> Email:  [hidden email]
>
> Web:   www.allanalog.com <http://www.allanalog.com/>  **
>
>  
>
>  
>



-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
<a href="http://ads.osdn.com/?ad_idt12&alloc_id344&op=click">http://ads.osdn.com/?ad_idt12&alloc_id344&op=click
_______________________________________________
Tikiwiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel