[Tiki-devel] Relaxing warning message

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Tiki-devel] Relaxing warning message

Bernard Sfez-3
Hello Dev,

I create a Tiki17 I using utf8mb4 (utf8mb4_unicode_ci) on local as it seems this is the right way to go now.

I got warning message on install.

———
Encoding Issue

Your database encoding is not in UTF-8.
Current encoding is utf8mb4. The languages that will be available for content on the site will be limited. If you plan on using languages not covered by the character set, you should re-create or alter the database so the default encoding is utf8.

More information

Character Set Conversion
Use at your own risk. If the data in the database currently contains improperly converted data, this may make matters worse. Suitable for new installations. Requires ALTER privilege on the database.
———

This is because : {if ($database_charset neq 'utf8' or isset($legacy_collation)) and $tikidb_created}
At : templates/tiki-install.tpl

Shouldn’t we relax the message for compatible encoding ? :)


Bernard Sfez | bsfez.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Relaxing warning message

Dr. Sassafras
It's currently not a comparable encoding. There is a few indexes that need to be refined before we're able to use it. After that I think the general consensus is that we move to it permanently. 

Brendan

On Jul 1, 2017, at 11:49 AM, Bernard Sfez <[hidden email]> wrote:

Hello Dev,

I create a Tiki17 I using utf8mb4 (utf8mb4_unicode_ci) on local as it seems this is the right way to go now.

I got warning message on install.

———
Encoding Issue

Your database encoding is not in UTF-8.
Current encoding is utf8mb4. The languages that will be available for content on the site will be limited. If you plan on using languages not covered by the character set, you should re-create or alter the database so the default encoding is utf8.

More information

Character Set Conversion
Use at your own risk. If the data in the database currently contains improperly converted data, this may make matters worse. Suitable for new installations. Requires ALTER privilege on the database.
———

This is because : {if ($database_charset neq 'utf8' or isset($legacy_collation)) and $tikidb_created}
At : templates/tiki-install.tpl

Shouldn’t we relax the message for compatible encoding ? :)


Bernard Sfez | bsfez.com

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Relaxing warning message

Bernard Sfez-3
Hi Brendan,

So far it is running fine, but I do believe you know more than me about it. :)

Will be happy if this can also move on… so we have it on Tiki18 ?

B


On 1 Jul 2017, at 21:59 , Dr. Sassafras <[hidden email]> wrote:

It's currently not a comparable encoding. There is a few indexes that need to be refined before we're able to use it. After that I think the general consensus is that we move to it permanently. 

Brendan

On Jul 1, 2017, at 11:49 AM, Bernard Sfez <[hidden email]> wrote:

Hello Dev,

I create a Tiki17 I using utf8mb4 (utf8mb4_unicode_ci) on local as it seems this is the right way to go now.

I got warning message on install.

———
Encoding Issue

Your database encoding is not in UTF-8.
Current encoding is utf8mb4. The languages that will be available for content on the site will be limited. If you plan on using languages not covered by the character set, you should re-create or alter the database so the default encoding is utf8.

More information

Character Set Conversion
Use at your own risk. If the data in the database currently contains improperly converted data, this may make matters worse. Suitable for new installations. Requires ALTER privilege on the database.
———

This is because : {if ($database_charset neq 'utf8' or isset($legacy_collation)) and $tikidb_created}
At : templates/tiki-install.tpl

Shouldn’t we relax the message for compatible encoding ? :)


Bernard Sfez | bsfez.com

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

Bernard Sfez | bsfez.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Relaxing warning message

Dr. Sassafras
Hmm. It probably needs a discussion. 

That encoding was introduced in MySQL 5.5.3. We currently support 5.0.... moving to mb4 encoding fixed some really bad errors (kill tiki without error if you save an emoji).

The maximum length of fixed byte types changes, so for example a tinytext is 255bytes. It's storage is reduced from 85 to 63 characters (difference between 3byte and 4byte storage)

Varchars are not effected. 

Index keys work in the same way, they are also a fixed byte type. 

Brendan

On Jul 2, 2017, at 1:12 AM, Bernard Sfez <[hidden email]> wrote:

Hi Brendan,

So far it is running fine, but I do believe you know more than me about it. :)

Will be happy if this can also move on… so we have it on Tiki18 ?

B


On 1 Jul 2017, at 21:59 , Dr. Sassafras <[hidden email]> wrote:

It's currently not a comparable encoding. There is a few indexes that need to be refined before we're able to use it. After that I think the general consensus is that we move to it permanently. 

Brendan

On Jul 1, 2017, at 11:49 AM, Bernard Sfez <[hidden email]> wrote:

Hello Dev,

I create a Tiki17 I using utf8mb4 (utf8mb4_unicode_ci) on local as it seems this is the right way to go now.

I got warning message on install.

———
Encoding Issue

Your database encoding is not in UTF-8.
Current encoding is utf8mb4. The languages that will be available for content on the site will be limited. If you plan on using languages not covered by the character set, you should re-create or alter the database so the default encoding is utf8.

More information

Character Set Conversion
Use at your own risk. If the data in the database currently contains improperly converted data, this may make matters worse. Suitable for new installations. Requires ALTER privilege on the database.
———

This is because : {if ($database_charset neq 'utf8' or isset($legacy_collation)) and $tikidb_created}
At : templates/tiki-install.tpl

Shouldn’t we relax the message for compatible encoding ? :)


Bernard Sfez | bsfez.com

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

Bernard Sfez | bsfez.com

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Tiki-devel] Relaxing warning message

Dr. Sassafras
In reply to this post by Bernard Sfez-3
Sorry, last one got away from me before I was done..

Index key lengths are also an issue InnoDB default 5.5) can support 3072 bytes in index keys but that wasn't made default until 5.7.7. Otherwise it will only support only 767 bytes. MyISAM on the other hand only supports 1000 bytes... so it's probably safest if we reduce index key lengths to 190 characters to stay under the 767 byte length.

There is also a small performance hit when upgrading. It would be nice if tiki could support multiple encoding types, so if I'm storing something that will only ever be in ascii (1byte), like a md5. I can specify ascii encoding type and it will improve tiki performance. Ucs2/UTF16 (2byte) introduces accents for European languages and more symbols. Then comes UTF8 (3byte) we're using now, it supports everything except:
  • Emojis
  • Mathematical/scientific symbols
  • Some Chinese, Japanese and Korean signs
  • Musical notation
  • Obscure/ancient languages
  • Emoticons, astral symbols, game symbols and other pictographic sets
  • (User defined) Font icons (these normally use reserved planes)
So if a field will never contain those things, the UTF8 encoding is still a little faster. 

All 4 of these encoding types are comparable. All the characters they support have the same hex value, so converting them poses little risk.

I think the only thing stopping us from supporting multiple encodings is the conversion process in the installer.

Moving forward:
*Redefine index key lengths (fairly easy and can be done now)
*Redefine fixed type vars (this one might be tricky. Say we have a tinytext declared.... does it need the full 85 characters or will 65 character limit suffice? If 66 characters are needed, that column type needs to be redefined) Ideally this would be done at the time of the switch, although doing it beforehand is also posable. 
* update minimum MySQL version to 5.5.3

So you can see there is room for errors while changing the encoding type here.

What are others thoughts? Is this something that should be done for tiki 19? We could strand people who are using post 5.5.3 in the Short Term  Support versions if we upgrade the requirement mid cycle. Is it worth stranding those people (probably very few of them) to fix this somewhat serious error in tiki. 

Brendan

On Jul 2, 2017, at 1:12 AM, Bernard Sfez <[hidden email]> wrote:

Hi Brendan,

So far it is running fine, but I do believe you know more than me about it. :)

Will be happy if this can also move on… so we have it on Tiki18 ?

B


On 1 Jul 2017, at 21:59 , Dr. Sassafras <[hidden email]> wrote:

It's currently not a comparable encoding. There is a few indexes that need to be refined before we're able to use it. After that I think the general consensus is that we move to it permanently. 

Brendan

On Jul 1, 2017, at 11:49 AM, Bernard Sfez <[hidden email]> wrote:

Hello Dev,

I create a Tiki17 I using utf8mb4 (utf8mb4_unicode_ci) on local as it seems this is the right way to go now.

I got warning message on install.

———
Encoding Issue

Your database encoding is not in UTF-8.
Current encoding is utf8mb4. The languages that will be available for content on the site will be limited. If you plan on using languages not covered by the character set, you should re-create or alter the database so the default encoding is utf8.

More information

Character Set Conversion
Use at your own risk. If the data in the database currently contains improperly converted data, this may make matters worse. Suitable for new installations. Requires ALTER privilege on the database.
———

This is because : {if ($database_charset neq 'utf8' or isset($legacy_collation)) and $tikidb_created}
At : templates/tiki-install.tpl

Shouldn’t we relax the message for compatible encoding ? :)


Bernard Sfez | bsfez.com

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

Bernard Sfez | bsfez.com

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
TikiWiki-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel