BigCommerce - Hosted Ecommerce Software by Interspire

Interspire Forum

 
Go Back   Interspire Forum > Interspire Email Marketer Community Forum > Customization and Integration

Reply
 
Thread Tools Display Modes
  #1  
Old 12-17-2010, 06:11 PM
James (Interspire Staff) James (Interspire Staff) is offline
Super Moderator
 
Join Date: Oct 2010
Posts: 14
Default Character Encoding Fixes

Hello everyone-

I have run into a few character-encoding issues in IEM recently, that I thought you might want to know about. We have fixes for these issues, which are already committed to the development server, and will be included in the next release of IEM. But if you are having problems related to character encoding, you can apply the fixes now. If you are using IEM in an English-only environment, these changes will have approximately zero impact on you. These issues are really only going to be apparent if you have non-ASCII content in your application.

I have attached a zip archive which contains a set of diff files which show the changes to be made. Here is a page which explains how to read diff files:

http://www.markusbe.com/2009/12/how-...a-unified-diff

The files to be updated are as follows:
[Your IEM Install directory]/admin/com/ext/database/mysql.php
[Your IEM Install directory]/admin/includes/js/javascript.js
[Your IEM Install directory]/admin/includes/config.php
[Your IEM Install directory]/admin/functions/settings.php

Now, a little background on the issues, for those interested. IEM is intended to be used with the UTF-8 character set, since this character set is quickly becoming the most common, and can easily handle characters from nearly any language on the planet. It's really a great thing, UTF-8.

However, in certain areas of the application, the code was not set to explicitly use UTF-8, and instead fell back to environment defaults. This is all supposed to be handled in the application by a single constant which stores the character to be used, but the constant was not being defined properly. In addition, the database API was doing a similar thing with its own character set value. If you want to get more in depth, you should probably just take a look at the attached diff files. There are also some bug reports you can read here:

https://track.interspire.com/browse/IEM-129
https://track.interspire.com/browse/IEM-259

All of this is fixed in the next version, which should be released in the near future.
Attached Files
File Type: zip charset-fix.zip (1.7 KB, 129 views)
__________________
Best regards,

James Adams
Support Engineer
Interspire Email Marketer
Interspire, Inc.

http://www.interspire.com/

Last edited by James (Interspire Staff); 12-22-2010 at 10:50 PM..
Reply With Quote
  #2  
Old 12-18-2010, 05:06 AM
urosg + urosg is offline
Interspire Customer
 
Join Date: Dec 2010
Posts: 2
Default mySQL Diff file incorrect

Hi James,

if I read the diffs correctly you have diffed the /admin/com/ext/database/mysql.php in the wrong direction and it shown to remove the new lines instead of adding them.

Kind regards,
Uros
Reply With Quote
  #3  
Old 12-20-2010, 03:22 PM
James (Interspire Staff) James (Interspire Staff) is offline
Super Moderator
 
Join Date: Oct 2010
Posts: 14
Default

Yikes! Good catch, urosg. Thanks for your help. I have re-diffed mysql.php, and updated the attachment in the original post.
__________________
Best regards,

James Adams
Support Engineer
Interspire Email Marketer
Interspire, Inc.

http://www.interspire.com/
Reply With Quote
  #4  
Old 05-13-2011, 08:51 AM
yusufdincer + yusufdincer is offline
Interspire Customer
 
Join Date: May 2011
Posts: 3
Default

There is a character problem while importing data from an external csv file. It does not import foreign characters like ş,i,ç,ö,ğ,ü,ı,İ. Can this modficatiın fix this problem?
Reply With Quote
  #5  
Old 05-14-2011, 08:22 AM
masilon + masilon is offline
Interspire Customer
 
Join Date: Jan 2004
Posts: 170
Default

I think that bug is still pending,see: https://track.interspire.com/browse/IEM-148
Reply With Quote
  #6  
Old 05-14-2011, 08:59 AM
Teneo + Teneo is offline
Interspire Customer
 
Join Date: Feb 2009
Location: Brugges
Posts: 630
Default

Hi James

We seem to run into issues with a customer trying to import French contact list.
The announcement field, like you would have "dear" in English is in french "Chère"

When this is tried to be imported the field just remains empty after importing from CSV.
Could this be fixed by this encoding fix ?

thank you very much !
__________________

PROMO : Signup today for our SMTP Relay services and receive a free license of our MPS Addon !

Bram Van Daele

TENEO SMTP relay services
http://esp.teneo.be

(Contact me directly)
Reply With Quote
  #7  
Old 05-14-2011, 01:17 PM
yusufdincer + yusufdincer is offline
Interspire Customer
 
Join Date: May 2011
Posts: 3
Default

Here is a reply from Marin-Interspire Staff

"Unfortunately the problem you are experiencing is caused by Microsoft Excel (judging from the *.csv you sent me, that's the program you used, right?) and partially by bad formatted data.

Let me explain you everything one by one.

Regarding bad formatted data - I see not all of your fields contain an email address - IEM won't let you import contacts whose email address fields are empty.

Regarding the special characters problem caused by Microsoft Excel, the workaround is the following:

- Open your Excel (*.xls) file
- Go to File -> Save As -> Unicode Text
- Open the exported Unicode Text (*.txt) file with an advanced text editor like Notepad++
- Go to Search -> Replace and replace all tabulations with ";" or ","
- Go to Encoding -> Convert to UTF-8 without BOM
- Go to File -> Save As and type a filename ending with the ".csv" extension
- Import the file into IEM

I know that's a bit of a headache, but that's the only way to import the special characters properly. It's a Microsoft issue that we can do nothing about...

Please try the workaround that I suggested and let me know if it helped...

Marin"

But I could not understand this line:

- Go to Search -> Replace and replace all tabulations with ";" or ","

What does it mean? do you have any idea ?

Thank you.
Reply With Quote
  #8  
Old 05-17-2011, 10:58 PM
James (Interspire Staff) James (Interspire Staff) is offline
Super Moderator
 
Join Date: Oct 2010
Posts: 14
Default

The suggestions quoted from Marin will resolve 99% of CSV import issues. If you still have trouble with this, please feel free to open a support ticket, or reopen the one it sounds like you already have, and we'll be happy to take a look at it further.
__________________
Best regards,

James Adams
Support Engineer
Interspire Email Marketer
Interspire, Inc.

http://www.interspire.com/
Reply With Quote
  #9  
Old 05-17-2011, 11:04 PM
James (Interspire Staff) James (Interspire Staff) is offline
Super Moderator
 
Join Date: Oct 2010
Posts: 14
Default

Oh, and the line you didn't understand:

- Go to Search -> Replace and replace all tabulations with ";" or ","

If you open the file in a text editor, they usually have a Find/Replace feature, much like a word processor. When he says "tabulations", I believe he is referring to the delimiters. You can really choose any character to separate values in a CSV, called the delimiter. The default would be the comma (hence the name Comma Separated Values), so you would have data like:

john@example.com,John,Smith
sam@example.com,Sam,Brown

But some CSV files might be formatted like this:

john@example.com%John%Smith
sam@example.com%Sam%Brown

or...

john@example.com|John|Smith
sam@example.com|Sam|Brown

which could cause problems in some cases. So Marin is suggesting you make sure the file is using either the comma (,) or semicolon ( as delimiter, to avoid unforeseen difficulties.
__________________
Best regards,

James Adams
Support Engineer
Interspire Email Marketer
Interspire, Inc.

http://www.interspire.com/
Reply With Quote
  #10  
Old 06-08-2011, 08:23 PM
claret
Guest
 
Posts: n/a
Default Error in diff paths...

Hi James,

I have just applied the suggested changes to my install. Everything seems stable after one logoff and login.

However, while applying the changes, I noticed the following error in your original post:

The changes required for settings.php should be applied to:
[Your IEM Install directory]/admin/functions/API/settings.php
rather than :
[Your IEM Install directory]/admin/functions/settings.php

Looking at the path in the diff file, it does point to the settings.php file under the api folder.

Please check it out and update your post if you deem my finding accurate.

Also, here's the approach I used to import my contacts with French characters from an excel spreadsheet:

- Open your Excel document containing your list
- Highlight and copy the range of cells to import
- Create a blank document With an advanced text editor like Notepad++
- Paste your list into it
- Go to Encoding -> Convert to UTF-8 without BOM
- Go to File -> Save As and type a filename (keep the ".txt" file-type)
- Import the file into IEM using "TAB" as the field separator

I prefer using tabs when importing and exporting data as I seldom find such characters in text fields as compared to commas.

Pierre
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On

Forum Jump