How connect scio scanner ?

[onoff0]

I have a SCIO scnner without a developer license, I'm following your project I launched "python read_spectrum.py" getting errors in file1.txt!
is there a guide on how to make SCIO work with your project?
thank you!

[kebasaa]

Hi onoff0, great that you found my project.
Essentially, I haven't written the python interface to read the data from the SCIO yet. I read some data "manually", which is in the examples folder, and "read_spectrum.py" will read those files and try to extract the spectrum from them. Unfortunately, the data is encoded (as you can see in the "readme" file), and I still haven't managed to decode it. I could use some help with that. Once I know how to decode that data (which is what I need help with), writing a bluetooth interface to make the SCIO work will be easy. I would really appreciate if you have ANY idea how to proceed

Now, in order to get read_spectrum.py to work with file1.txt, I'd need to know a few more details of what exactly your errors are. Also, try with file2.txt or something like that.

[onoff0]

to have the clearest concept: the raw data where you took them (file1 | 2.txt)? not having the developer license do you have any other solution or idea to decode the data in the test files? Excuse me for my English, but I need these questions to better understand the work you are doing Thanks ! Il Mar 9 Lug 2019, 08:35 kebasaa <notifications@github.com> ha scritto:

…

Hi onoff0, great that you found my project. Essentially, I haven't written the python interface to read the data from the SCIO yet. I read some data "manually", which is in the examples folder, and "read_spectrum.py" will read those files and try to extract the spectrum from them. Unfortunately, the data is encoded (as you can see in the "readme" file), and I still haven't managed to decode it. I could use some help with that. Once I know how to decode that data (which is what I need help with), writing a bluetooth interface to make the SCIO work will be easy. I would really appreciate if you have ANY idea how to proceed Now, in order to get read_spectrum.py to work with file1.txt, I'd need to know a few more details of what exactly your errors are. Also, try with file2.txt or something like that. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFOLXD6QBFNUIP4UDYLP6VDP6QWUJA5CNFSM4H63PAIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZPH7MQ#issuecomment-509509554>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFOLXD34EH7T5MCGFNUFUSTP6QWUJANCNFSM4H63PAIA> .

[kebasaa]

Hi,

No problem, I am happy to explain.

I don't have a developer license. Therefore, I used bluetooth to collect the encoded raw data. Now I need to decode it. For this, I need to understand what the structure of the data is.

If you want to collect your own raw data from the SCIO, here is a step by step instruction:

1. On Linux, install gatttool and hcitool. I'm using Ubuntu, to install:

    sudo apt-get install bluez

2. Turn on your SCIO with a long press on the button

3. Run hcitool to find out what your SCIO's MAC address is. It will have a name like SCiOmyScio or whatever you named it:

    sudo hcitool lescan

4. Run gatttool with your SCIO's MAC address to collect your own data. This will store it in "file1.txt". Replace xx:xx:xx:xx:xx:xx with the MAC address you found in step 3. During the scan, the SCIO indicator light will be yellow.

    sudo gatttool -i hci0 -b xx:xx:xx:xx:xx:xx --char-write-req -a 0x0029 -n 01ba020000 --listen > file1.txt

5. Stop saving data to your file with Ctrl+C after the indicator light of the SCIO goes back to blue.

6. In a text editor, edit your file1.txt: Remove the first line saying "Characteristic value was written successfully" and in the beginning of each line remove "Notification handle = 0x0025 value: ". Then save the file

[kebasaa]

The main problem now is to take this series of hex values, and to understand what they mean, i.e. how to extract the spectrum from them. This is what I need help with. It turns out that the first hex value of each line identifies the line, i.e. 01 is the first line, etc. Values go from 01 to 5f and repeat 3 times, although for the 3rd scan, they go from 01 to 58 only

[onoff0]

ok now everything is clearer! but can these hexadecimal values ​​be decoded with any developer license? or is the license unique only on the device that requires the license? in both cases how do you plan to proceed? is there an algorithm that you have identified to work the raw data? I believe that their servers through the univocal developer license (user that requires it and the SCIO hardware) send a decryption key to the user, but it is only my thought, it could be checked on the same material with two different SCIOs if the raw data is the same, doing some diffing! What do you think to do ? A greeting! Il Mar 9 Lug 2019, 14:24 kebasaa <notifications@github.com> ha scritto:

…

The main problem now is to take this series of hex values, and to understand what they mean, i.e. how to extract the spectrum from them. This is what I need help with. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFOLXD3KKTZJVEEE6J5L5CLP6R7QJA5CNFSM4H63PAIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZQDEOY#issuecomment-509620795>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFOLXD3PQP375BDTVTSOJ5TP6R7QJANCNFSM4H63PAIA> .

[kebasaa]

Hi

Well, the developer license basically allows the android app to export a CSV or JSON file, which the server decoded from the raw hexadecimal values. The problem is that we don't know how the server decodes it, and so I thought that if I have a copy of raw hexadecimal data and of a CSV, I could try to figure it out.

What I know from people online is that the CSV or JSON that is exported supposedly contains 400 values of reflectance for each band from 700nm to 1100nm. After pre-processing the hexadecimal values, for each scans, there are 3 readings: Two times 1800 hexadecimal values, and one time 1656 hex values.

I do not think that the hexadecimal data is encrypted because the microprocessor inside the SCIO is quite weak. But the data is encoded. So the server does not send a decryption key, it only does some mathematics with the 3 readings of each scan. I think the second or third reading is related to calibration.

Do you have any idea how to proceed?

[onoff0]

the only idea that comes to mind is the same! ie remedy a decoded file to see which mathematical operators use the SCIO server! Otherwise knowing in principle the values ​​that it returns use a software that forces the data to obtain similar and desired values! like John the ripper (which was used for other purposes but the meaning is that) if I can do it tonight or tomorrow I want to try to see if I find something could be useful! What do you think? Il Mer 10 Lug 2019, 13:17 kebasaa <notifications@github.com> ha scritto:

…

Hi Well, the developer license basically allows the android app to export a CSV or JSON file, which the server decoded from the raw hexadecimal values. The problem is that we don't know how the server decodes it, and so I thought that if I have a copy of raw hexadecimal data and of a CSV, I could try to figure it out. What I know from people online is that the CSV or JSON that is exported supposedly contains 400 values of reflectance for each band from 700nm to 1100nm. After pre-processing the hexadecimal values, for each scans, there are 3 readings: Two times 1800 hexadecimal values, and one time 1656 hex values. I do not think that the hexadecimal data is encrypted because the microprocessor inside the SCIO is quite weak. But the data is encoded. So the server does not send a decryption key, it only does some mathematics with the 3 readings of each scan. I think the second or third reading is related to calibration. Do you have any idea how to proceed? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFOLXD2QHFSKK4SYZFRJNEDP6XAOJA5CNFSM4H63PAIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZTEQNQ#issuecomment-510019638>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFOLXD7AJ4GU4SANW5DOACDP6XAOJANCNFSM4H63PAIA> .

[franklin02]

Please provide your email address. I can send you a spectral data (CSV file).

[kebasaa]

Please provide your email address. I can send you a spectral data (CSV file).

@franklin02 I sent you an email. Will put your files in the example-data folder as soon as I have them. Thanks a lot!!!

[kebasaa]

@franklin02 Thanks a lot for the file, I added it to the example-data folder. Unfortunately, this is not what the SCIO with a developer license creates (that is supposed to include the spectrum, raw irradiance of the sensor and calibration data). Rather, the file you sent is a collection of scans of different materials, containing only the reflectance spectra. I have no idea how useful it will be....

[kebasaa]

@onoff0 I updated the script to properly read the scio raw data. The decoding is still a bit unclear though, but I now know that 5 hex values are one 1nm band. The conversion to a float is still a bit unclear...

[onoff0]

great ... can I know how did you identify that 1nm band corresponds to 5 hex value? what tools did you use? or what kind of algorithm did you use? other thing, I could have your private email thanks !!! Il Lun 15 Lug 2019, 22:32 kebasaa <notifications@github.com> ha scritto:

…

@onoff0 <https://github.com/onoff0> I updated the script to properly read the scio raw data. The decoding is still a bit unclear though, but I now know that 5 hex values are one 1nm band. The conversion to a float is still a bit unclear... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFOLXD4STD434NZWSTIIBQTP7TNGTA5CNFSM4H63PAIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ64H5Y#issuecomment-511558647>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFOLXD4YIOEEI6LF2UAYSGTP7TNGTANCNFSM4H63PAIA> .

[kebasaa]

It's an educated guess: The data comes in 3 parts. Each part has a header that contains some sort of identifier. After that, there are 1800, 1800 and 1656 hex values left. Knowing that we have 1nm bands, and 331 of them (counting from 740 to 1070nm). So I tried to divide 1800 by 4, 5 or 6 and never had success. Then, I divided 1656 by 331 and I got 5, with 1 hex value left (I assume it's a divider or something). Now, it makes sense that all the 3 parts contain the same data structure. So I did 1800 - 1656 and I got 144, which is a very nice round number indicating that it's a header or something. This means that either the data of the 2 first parts (with 1800 hex values) are either preceded or followed by a block of 144 values. I don't know which one yet.

[kebasaa]

@onoff0 Did you receive my e-mail address?

[onoff0]

yes I wrote you ... thanks! Il Mar 16 Lug 2019, 17:19 kebasaa <notifications@github.com> ha scritto:

…

@onoff0 <https://github.com/onoff0> Did you receive my e-mail address? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFOLXD23ZTVQNFEKLWM66TDP7XRJXA5CNFSM4H63PAIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2BGO4Y#issuecomment-511862643>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFOLXD4XKZECIVPNQA77X43P7XRJXANCNFSM4H63PAIA> .