Imports quick steps

From Cyclos4 Wiki
Jump to: navigation, search

Introduction

The import functionality can import the following (user) information:

  • Users together with all profile fields, addresses, phones and profile images;
  • Advertisements;
  • User records;
  • References;
  • Transfers;
  • Payments;
  • Tokens.

Importing works fairly easy. Go to system > tools > imports and click on import and select the appropriate data to import, if anything is missing you probably need to give your group the permission to import this data. Subsequently you can upload the csv file and click on save. Then you can click on a new tab called lines where you can decided not to import a line or to change (and correct) it. Finally you can click on the button "Import all results with status ready" to start the import process. Below you will find a detailed description of all fields that can be imported together with a lot of examples.


Be aware: transfers and payments can be imported in two ways. Through system > tools > imports and trough banking > tools > import payments. When importing payments through banking all checks are made that are normally done when a payment is made and the payment is always processed at the current date and time. When importing transfers through the system imports some settings and permissions are not taken into account. No fees are charged on imported transfers, no check is made if the user has sufficient balance, also it isn't checked if the user has the permission to pay the other user using the transfer type. This will make it much easier for admins to migrate to Cyclos, without reconfiguring the whole system after the imports have been made. We recommend to only enable the import of transfers in the test instance and when setting up the system, and disable the functionality as soon as the system goes live.


For a quick start you can copy one of the examples and start importing, if you want to now exactly how things work we recommend you to study the general rules and the rules per header.

Rules CSV or zip file

Rules

The data can be imported using a CSV file:

  • The first line of the file has to contain all the headers separated by the configured separator (a comma by default).
  • The separator is taken from the configuration of the administrator (Configuration: Localization > List separator).
  • Each of the following lines of the csv represents an user, ad, user record, reference, transfer, payment or token.
  • All lines need to have the same amount of fields (thus the same amount of separators).
  • Any filename can be used for the csv file as long as it has the csv filename extension;
  • A field or a line can be left blank.
  • It is recommended to place all fields between double quotes, this is recommended because in this case the separator can be used in fields without causing problems. So if you have the comma as the value separator, and you have the text: Hi, how are you doing? You should wrap it like this "Hi, how are you doing?" to prevent the parser from splitting it in two values.
  • When using the internal name of a field, make sure to write the internal name exactly the same as in Cyclos, the import is case sensitive.

Images

A zip file can be used to wrap the csv file and (optionally) profile images:

  • The zip file should contain only one csv file;
  • Any filename can be used for the zip file as long as it has the zip filename extension;
  • The zip file may contain multiple image files that can be placed inside any folder within the zip.
  • For images the header "images" can be used, and the field should contain the path of the image, e.g.: "images/image1.png, images/image2.png" or "image2.png".
  • If you want to upload multiple(profile or advertisement images) images, the images can be separated with a comma, e.g.: "images/image1.png, images/image2.png" or "image2.png". Makes sure to use double quotes!

Custom fields

  • The header is the concatenation of the string "field." and the internal name of the field, e.g. "field.remarkdate".
  • The values for fields of type list (enumerated) should match the corresponding possible values declared in the field. If a value doesn't match a possible value the import line will be marked with a validation error.
  • When importing multiple selection fields it is possible to import multiple values, by separating the values with a comma. Make sure to place double quotes around all values, e.g.: "value1,value2,value6" or "option1, option5".
  • When importing a boolean the possible values are "true" and "false", when the field is left blank the default value will be used.

Dates

  • Dates and date times are parsed according to ISO 8601 or according to the configuration properties for date and time format.

For example, it is possible to import an ISO date 1994-11-05T08:15:30-05:00 or import 05/11/1994 08:15 according to the following configuration: Date format "day/month/year" and time format "24 hours"

Import process

Step 1 (required): Reading files and validating

The administrator job:

  • Select the destination group and mark if necessary to send the activation email(for user imports), select the currency (for advertisements imports) or select the record type (for records imports). The other imports have no specific fields.
  • Write the import description.
  • Select the file (csv or zip) to upload and click save.


The system job:

  • The import will begin reading a zip file or a csv file.
    • If there are problems reading the file, corrupt or has invalid separator, then the import is marked with an 'Invalid' status and a message will be shown describing the error.
    • If a zip file was uploaded then the first state of the import is 'Reading images'. After reading all the images, the import switch to the next state 'Reading csv'.
    • If a csv file was uploaded then the first state of the import is 'Reading csv'. In this state, each line of the csv is parsed and the resulting entity is subject to a preliminary validation.
      • If a line passes the validation its status is set to 'Ready'.
      • If the line has a validation error its status will be 'Validation error' and an the associated error message will be shown.
      • If the number of fields in the line is not the expected then its status is turned to 'Invalid'.
      • If any other strange error occurs while validating the line, the line is marked with 'Invalid' and an internal error message is shown.
  • After the system has finished validating all the lines, the import status turns Ready.

Step 2 (optional): Skipping lines

When the import status is 'Ready' (to be imported), we can still make some changes. For example we can select specific lines with the status ready and mark then to be skipped.

Step 3 (optional): Editing lines

When the import is on the 'Ready' status, we can go to a particular line of the import and edit its content, just like if we were changing the csv file but without the need to upload it all over again. For example, if a particular line had a validation error because of the content of one field, we can edit this field of the line and save it with the correct value. The line is submitted to validation and if it succeeds the status will change to 'Ready'.

Step 4 (required): Importing lines

Once the import is in Ready state, it can be imported by clicking at the import button. At this point the import state will change to 'Importing'. The lines will change its status to Imported or Import error or Internal error (in case of a strange error). If the line status changes to Imported then the entity was successfully imported. If the line status changes to Import error, then an error message can be found with the cause of the failure. If the line status changes to Internal error, we should probably check the server logs to identify the problem.

When the import finishes processing all the lines the status will change to Imported or Internal error (in case of a strange error).

Step 5: Archive import

After 30 days, a background task will archive the imports in this way:

  • If the import status is not 'Imported', it will be deleted.
  • If the import status is 'Imported', then it will eliminate all the import information leaving the minimal following data:
    • Import description
    • Import creation and processing dates.
    • Import statistics (number of imported entities etc..)

User and accounts import

Headers

Header name Required Comment
Basic user fields
name Required The user's full name.
username Required The user's login name. The username must be unique, an error will be given when the name already exists in the system. In case the login name is an automatically generated (account)number the column is not needed and Cyclos will generate the login names.
email Required or optional according to the Cyclos setting 'Require e-mail'. It must be a well formatted e-mail address. If e-mail validation is enabled for importing users (in Configuration 'Validate e-mail on' - Import users) the users will get get the 'pending' status. Be aware that users with pending status cannot have an account, therefore in this case account information is not imported.
creationdate Optional If specified the date in the personal member reports (member since) will be set with this date. If no date is set in the column the current (import) date will be set. Check the configuration to see what date format is expected.
broker Optional If a broker is specified, it will be assigned as the main broker. Only active brokers are valid.
images Optional For profile images the header "images" can be used, and the field should contain the path of the images, for more information see #Images.
Address fields
  • In the configuration is set if an address is required.
  • If you want to import more then one address per user an identifier can be used, it should be a unique name e.g.: "address[1].name","address[2].name".
  • If only one address is imported the identifier can be skipped, e.g.: "address.line1".
  • Be aware: the address will not appear right away on the Google map, but a polling tasks executes this task one by one (which might take some time).
address[identifier].name Optional The name of the address, when no name is given the address will get the name "address 1", "address 2", etc..
address[identifier].line1 Depends on configuration (required address fields) The first line of the address (typically street name).
address[identifier].line2 Depends on configuration (required address fields) Second line of the address.
address[identifier].neighborhood Depends on configuration (required address fields) Neighborhood of the address.
address[identifier].pobox Depends on configuration (required address fields) PO Box identification.
address[identifier].zip Depends on configuration (required address fields) ZIP code of the address
address[identifier].city Depends on configuration (required address fields) The city of the address. When it is not specified and the default city is specified in the configuration, the default city will be used.
address[identifier].region Depends on configuration (required address fields) The region or state of the address. When it is not specified and the default region or state is specified in the configuration, the default region or state will be used.
address[identifier].country Depends on configuration (required address fields) 2-letter country code defined in ISO 3166, (must be in capital characters). When it is not specified and the default country is specified in the configuration, the default country will be used.
address[identifier].private Optional When the address is set to private other users cannot see this address. When it is not specified the default settings from the configuration are used. Possible options are "yes", "true", "no" and "false".
address[identifier].buildingnumber Depends on configuration (required address fields) The building number.
address[identifier].complement Depends on configuration (required address fields) The address complement field.
address[identifier].street Depends on configuration (required address fields) The name of the street.


Phone number
  • In the configuration is set if an mobile of or landline phone number is required.
  • If you want to import more then one phone number per user an identifier can be used, it should be a unique name e.g.: "phone[1].name","phone[2].name".
  • If only one phone number is imported the identifier can be skipped, e.g.: "mobile.number".
mobile[identifier].name Optional Name for the mobile phone number (e.g. private).
mobile[identifier].number Depends on configuration The mobile number.
mobile[identifier].private Optional When the mobile number is set to private other users cannot see this number. When it is not specified the default settings from the configuration are used. Possible options are "yes", "true", "no" and "false".
landline[identifier].name Optional A name for the land line phone number (e.g. "home").
landline[identifier].number Depends on configuration The land line number.
landline[identifier].private Optional When the landline number is set to private other users cannot see this number. When it is not specified the default settings from the configuration are used. Possible options are "yes", "true", "no" and "false".
Profile fields (custom)
  • Check the products associated to the users group to know which custom fields can be imported. The ones that can be imported are those selected under the column "Enabled" of the "My profile fields" permission.
field.<internalname> Depends on field settings. The user profile field, for more information see #Custom_fields.
field.<internalname>.private Optional When the custom field is set to private other users cannot see it for this user. When it is not specified ??????????. Possible options are "yes", "true", "no" and "false".
Account fields
  • Some configurations can be specified for each user account.
account.<internalname>.number Optional The number for the specified account. In case the user remains pending for activation, the account number is reserved until the user is activated or removed.
account.<internalname>.negativelimit Optional The negative limit for the specified account. If empty or not specified the limit will follow what is specified in the products.
account.<internalname>.positivelimit Optional The positive limit for the specified account. If empty or not specified the limit will follow what is specified in the products. UNLIMITED or unlimited can be specified to remove the upper limit.
Password fields
password.<internalname>.value or password Required or optional according to the Cyclos setting in password type (default password type is manual). The generated passwords (i.e not manual) will not be created if "Send activation email" is not checked, there's a bulk action you can execute to generate and send the passwords to the users. A password is required if it's selected at any channel configuration, it's marked 'At registration' in the products of the importing group and "Send activation email" is not checked. If only one password type is marked 'At registration' you can eventually specify this column as password. Be aware: if the the password you are trying to import is not used in any channel configuration for the user you are trying to import, then the password is ignored.
password.<internalname>.forcechange or password.forcechange Optional Forces the user to change the password upon login. If only one password type is marked 'At registration' you can eventually specify this column as password.forcechange.



Example 1: most simple cases

"name","username","email","password.login.value"
"user1","user1","user1@test.com","1234"
"user2","user2","user2@test.com","1234"
"user3","user3","user3@test.com","1234"
"user4","user4","user4@test.com","1234"
"user5","user5","user5@test.com","1234"
"user6","user6","user6@test.com","1234"

Note:

  • In the configuration of this example the list separator is a comma.
  • Login password has internal name: login

Example 2: multiple addresses per user

"name","username","email","password.login.value","address[1].name","address[1].line1","address[1].city","address[1].country","address[2].name","address[2].line1","address[2].city","address[2].country"
"Paul McCartney","mccartney","Paul@McCartney.com","1234","Work","69 Royal Worcester Drive","London","GB","Home","Albert Dock","Liverpool","GB"
"John Lennon","john_lennon","John@JohnLennon.com","1234","Work","70 Royal Worcester Drive","London","GB","Home","Albert Dock","Liverpool","GB"

Note:

  • In the configuration of this example the list separator is a comma.
  • Login password has internal name: login
  • Be aware that the addresses will not appear on google map directly, this will be done trough time by a scheduled task, in order to spread the load on google maps and stay within the limits.

Example 3: multiple phone numbers per user

"name","username","email","password.login.value","creationdate","mobile[1].name","mobile[1].number","mobile[2].name","mobile[2].number","landline.name","landline.number"
"Paul McCartney","mccartney","Paul@McCartney.com","1234","25-08-2000","Private","07123456781","Work","07123456782","House","02012345671"
"John Lennon","john_lennon","John@JohnLennon.com","1234","01-01-2001","Private","07123456783","Work","07123456784","House","02012345672"

Note:

  • In the configuration of this example the list separator is a comma.
  • The date format is "day-month-year".
  • The localization country is the United Kingdom.

Example 4: users with profile fields

"name","username","email","password.login.value","field.birthday","field.gender"
"Paul McCartney","mccartney","Paul@McCartney.com","1234","25-08-1982","male"
"John Lennon","john_lennon","John@JohnLennon.com","1234","19-12-1964","male"
"user1","user1","user1@test.com","1234","19-12-1964","male"
"user2","user2","user2@test.com","1234","19-12-1964","female"
"user3","user3","user3@test.com","1234","19-12-1964","male"
"user4","user4","user4@test.com","1234","19-12-1964","female"
"user5","user5","user5@test.com","1234","19-12-1964","male"
"user6","user6","user6@test.com","1234","19-12-1964","female"

Note:

  • In the configuration of this example the list separator is a comma.
  • Login password has internal name: login
  • The date format is day-month-year.
  • There is a profile fields with the internal name birthday which is date field.
  • There is another profile field with the internal name gender which is an enumerated field with the possible values male and female.

Example 5: users with profile images

Please download the zip file: http://www.cyclos.org/wiki4/files/users.zip

Note:

  • In the configuration of this example the list separator is a comma.
  • In the configuration the email is set to not be required.

Example 6: users with account configuration

"name";"username";"email";"password.login.value";"account.units.number";"account.units.negativelimit";"account.units.positivelimit"
"user1";"user1";"user1@test.com";"1234";"0000-0001";"1000";"UNLIMITED"
"user2";"user2";"user2@test.com";"1234";"0000-0002";"1000";"100000"

Advertisment import

Headers

Header name Required Comment
Basic ad fields
user Required The identification value of the user that will own the advertisement.
title Required The title of the advertisement.
description Required The description of the advertisement.
categories Required A comma separated list of categories of the advertisement, must exactly represent the complete path of internal names. e.g. "services>household>cleaning, paymentMethod>cashOnly".
creationdate Optional The creation date of the advertisement, when it is not specified the current date will be used as creation date.
publicationbegin Optional The day the advertisement will be published and thus visible for other users, when it is not specified the creation date will be used as begin date.
publicationend Optional The day the publication of advertisement will end and thus not be visible anymore for other users, when it is not specified, the date will be calculated using the default publication period set in products.
price Optional The price of the advertisement, please use the decimal separator as configured in the number format of the configuration.
promotionalprice Optional The promotional price of the advertisement.
promotionalperiodbegin Required if promotional price is set The date the promotion starts and the promotional price is used.
promotionalperiodend Required if promotional price is set The date the promotion ends and the promotional price is not used anymore.
hidden Optional A hidden advertisement can't be seen by other users. Options: true / false. When the field is not specified the advertisement will not be hidden by default.
images Optional For ad images the header "images" can be used, and the field should contain the path of the images, for more information see #Images.
Advertisement fields (custom)
  • Check the products associated to the users group to know which fields can be imported. The ones that can be imported are those selected in "Enable ad fields".
field.<internalname> Depends on field settings. The advertisement field. For more information see #Custom_fields.


Example 1: basic ads import

"user","title","description","categories","price"
"user1","Car","Nice car, owned by an old lady.","goods>car","1399.99"
"user2","Cat","I need to find a new owner for my cat, because he is trying to eat my mouse.","animals>small","100"
"user2","Mouse","I need to find a new owner for my mouse, because my cat doesn't like him","animals>small","55"
"user3","Bike","Brand new bike, never used.","goods>bikes","150"
"user4","Jeans","Used jeans for sale, only has a few holes, but thats popular nowadays.","goods>clothing>other","29.99"
"user4","Shoes","Nice nike air max.","goods>clothing>shoes","59"

Note:

  • Make sure the users exist.
  • In this example the decimal separator is a dot.
  • Ads can only be created in a leaf category. So if "goods>clothing>shoes" exist, you cannot publish anything in "goods>clothing".

User records import

Headers

Header name Required Comment
Basic record fields
user Required The identification value of the user who will get the user record.
date Optional The date the user record is created, if left empty the current date will be used.
createdby Optional The identification value of the user who created (filled in) the user record, if left empty the current user that commits the import will be used.
lastmodificationdate Optional The last date the user record has been changed (saved), if left empty the user record will not display this field.
lastmodifiedby Optional The identification value of the last user that has changed (saved) the user record, if left empty the user record will not display this field.
Record type custom fields
field.<internalname> Depends on field settings. The record fields, for more information see #Custom_fields.


Example 1: full import

"user","date","createdby","lastmodificationdate","lastmodifiedby","field.comments"
"user1","16-06-2014 00:00","admin","18-06-2014 00:00","admin","An user record which has a single text custom field."

Note:

  • The record type can be selected after uploading the file.
  • Make sure the users exist also the createdby user.
  • The date format is day-month-year.

Example 2: simple import

"user","field.comments"
"user1","An user record which has a multi line text field."

Note:

  • The record type can be selected after uploading the file.
  • Make sure the user1 exists.

References import

Headers

Header name Required Comment
from Required The username of the user that will give the reference.
to Required The username of the user that will receive the reference.
level Required The level (appreciation) of the user, options:
  • VERY_GOOD
  • GOOD
  • NEUTRAL
  • BAD
  • VERY_BAD
comments Required The comment that can be given together with the reference.
date Optional The date the reference is given, if left empty the current date will be used.


Example 1: reference import

"from","to","level","comments"
"user1","user6","VERY_GOOD","Good service, thanks!"
"user2","user6","GOOD","This guy is the best :)"
"user3","user6","NEUTRAL","Alright..."
"user4","user6","BAD","He should improve."
"user5","user6","VERY_BAD","Don't trust him!"

Note:

  • In the configuration of this example the list separator is a comma.

Payment import

Payments are imported via Banking > Tools > Imports.

Headers

Header name Required Comment
Basic payment fields
from Required The user who will send the payment, for the user use the username for the system use "system".
to Required The user who will receive the payment, for the user use the username for the system use "system".
type Required The transfer type that will be used for the payment.
  • Must be in the format: <account internal name>.<tranfer type internal name>
amount Required The amount that will be transferred.
description Optional The description of the payment.
status.<flowInternalName> Optional The status of the payment, the internal neame of the flow must be used.
Transfer type payment fields
field.<internalname> Depends on field settings The (custom) payment fields.


Example: simple payment import

"from","to","type","amount","description","status.flow1"
"user1","user2","unitsaccount.tradetransferunits","2","Test 2",""
"user1","user3","unitsaccount.tradetransferunits","3","Test 3",""
"user1","user4","unitsaccount.tradetransferunits","4","Test 4",""
"user1","user5","unitsaccount.tradetransferunits","5","Test 5",""
"user1","user6","unitsaccount.tradetransferunits","6","Test 6",""

Note:

  • Make sure there is an account with the internal name "unitsaccount".
  • Make sure there is a transfer type with the internal name "tradetransferunits".
  • Make sure the date format is "day/month/year".
  • Make sure the user1 to user 6 exists.

Transfer import

Transfers are imported via System > Tools > Imports. Be aware: when importing transfers via system imports fees are ignored, the balance of the user is not checked and the user doesn't need the permission to do the payment.

Headers

Header name Required Comment
Basic transfer fields
from Required The user who will send the payment, for the user use the username for the system use "system".
to Required The user who will receive the payment, for the user use the username for the system use "system".
type Required The transfer type that will be used for the payment.
  • Must be in the format: <account internal name>.<tranfer type internal name>
date Optional When it is not specified the current date will be used as creation date.
amount Required The amount that will be transferred.
description Optional The description of the payment.
status.<flowInternalName> Optional The status of the payment, the internal neame of the flow must be used.
Transfer type payment fields
field.<internalname> Depends on field settings The (custom) payment fields.


Example: simple transfer import

"from","to","type","date","amount","description","status.flow1"
"user1","user2","unitsaccount.tradetransferunits","12/06/2014 00:00","2","Test 2",""
"user1","user3","unitsaccount.tradetransferunits","14/06/2014 00:00","3","Test 3",""
"user1","user4","unitsaccount.tradetransferunits","13/06/2014 00:00","4","Test 4",""
"user1","user5","unitsaccount.tradetransferunits","16/06/2014 00:00","5","Test 5",""
"user1","user6","unitsaccount.tradetransferunits","19/06/2014 00:00","6","Test 6",""

Note:

  • Make sure there is an account with the internal name "unitsaccount".
  • Make sure there is a transfer type with the internal name "tradetransferunits".
  • Make sure the date format is "day/month/year".
  • Make sure the user1 to user 6 exists.

Token import

Headers

Header name Required Comment
Basic token fields
user Optional The username of the user to be assigned to the token
type Required The internal name of the token type
activatenow Optional Boolean value to indicate if the token should be activated or not.
value Required The token value
expiry Optional The token expiry date.
activationdeadline Optional The token activation deadline.


Example 1: minimal token import fields

After import the tokens they will be in Unassigned status and users with properly permission can activate them.

"type","value"
"visa","1234123412341234"
"master","1234123412341235"

Example 2: full token import

"user","type","activatenow","value","expiry","activationdeadline"
"user1","visa","true","1234123412341234","02/07/2060",""
"user1","master","false","1234123412341235","","02/07/2060"