The High-Level Practical Overview of Open-Source Privacy-Preserving Machine Learning Solutions

Authors

  • Konrad Kuźniewski Intel Corporation
  • Krystian Matusiewicz Intel Corporation
  • Piotr Sapiecha Intel Corporation

Abstract

This paper aims to provide a high-level overview of practical approaches to machine-learning respecting the privacy and confidentiality of customer information, which is called Privacy-Preserving Machine Learning. First, the security approaches in offline-learning privacy methods are assessed. Those focused on modern cryptographic methods, such as Homomorphic Encryption and Secure Multi-Party Computation, as well as on dedicated combined hardware and software platforms like Trusted Execution Environment - Intel® Software Guard Extensions (Intel® SGX). Combining the security approaches with different machine learning architectures leads to our Proof of Concept in which the accuracy and speed of the security solutions will be examined. The next step was exploring and comparing the Open-Source Python-based solutions for PPML. Four solutions were selected from almost 40 separate, state-of-the-art systems: SyMPC, TF-Encrypted, TenSEAL, and Gramine. Three different Neural Network architectures were designed to show different libraries’ capabilities. The POC solves the image classification problem based on the MNIST dataset. As the computational results show, the accuracy of all considered secure approaches is similar. The maximum difference between non-secure and secure flow does not exceed 1.2%. In terms of secure computations, the most effective Privacy-Preserving Machine Learning library is based on Trusted Execution Environment, followed by Secure Multi-Party Computation and Homomorphic Encryption. However, most of those are at least 1000 times slower than the non-secure evaluation. Unfortunately, it is not acceptable for a real-world scenario. Future work could combine different security approaches, explore other new and existing state-of-the-art libraries or implement support for hardware-accelerated secure computation.

Author Biographies

Konrad Kuźniewski, Intel Corporation

Please include one bio in spot "Author X":

\author{Konrad~Kuźniewski, Krystian~Matusiewicz, Piotr~Sapiecha% <-this % stops a space    \thanks{Konrad~Kuźniewski, Krystian~Matusiewicz, Piotr~Sapiecha are working at Intel in the IPAS division. (e-mail: \{konrad.kuzniewski, krystian.matusiewicz, piotr.sapiecha\}@intel.com.}% <-this % stops a space}

Krystian Matusiewicz, Intel Corporation

Please include one bio in spot "Author X":

\author{Konrad~Kuźniewski, Krystian~Matusiewicz, Piotr~Sapiecha% <-this % stops a space    \thanks{Konrad~Kuźniewski, Krystian~Matusiewicz, Piotr~Sapiecha are working at Intel in the IPAS division. (e-mail: \{konrad.kuzniewski, krystian.matusiewicz, piotr.sapiecha\}@intel.com.}% <-this % stops a space}

Piotr Sapiecha, Intel Corporation

Please include one bio in spot "Author X":

\author{Konrad~Kuźniewski, Krystian~Matusiewicz, Piotr~Sapiecha% <-this % stops a space    \thanks{Konrad~Kuźniewski, Krystian~Matusiewicz, Piotr~Sapiecha are working at Intel in the IPAS division. (e-mail: \{konrad.kuzniewski, krystian.matusiewicz, piotr.sapiecha\}@intel.com.}% <-this % stops a space}

References

F. Newsroom. (2018) Fda permits marketing of artificial intelligence-

based device to detect certain diabetes-related eye problems. [Online].

Available: https://www.fda.gov/news-events/press-announcements/fda-

permits-marketing-artificial-intelligence-based-device-detect-certain-

diabetes-related-eye

FDA. Artificial intelligence and machine learning (ai/ml)-enabled

medical devices. [Online]. Available: https://www.fda.gov/medical-

devices/software-medical-device-samd/artificial-intelligence-and-

machine-learning-aiml-enabled-medical-devices

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J.

Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in

nd International Conference on Learning Representations, ICLR 2014,

Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings,

Y. Bengio and Y. LeCun, Eds., 2014. [Online]. Available: http:

//arxiv.org/abs/1312.6199

A. Shafahi, W. R. Huang, M. Najibi, O. Suciu, C. Studer,

T. Dumitras, and T. Goldstein, “Poison frogs! targeted clean-

label poisoning attacks on neural networks,” in Advances

in Neural Information Processing Systems 31: Annual Conference on

Neural Information Processing Systems 2018, NeurIPS 2018, December

-8, 2018, Montr ́eal, Canada, S. Bengio, H. M. Wallach, H. Larochelle,

K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., 2018, pp.

–6116. [Online]. Available: https://proceedings.neurips.cc/paper/

/hash/22722a343513ed45f14905eb07621686-Abstract.html

X. Yuan, P. He, Q. Zhu, and X. Li, “Adversarial examples: Attacks

and defenses for deep learning,” IEEE Trans. Neural Networks Learn.

Syst., vol. 30, no. 9, pp. 2805–2824, 2019. [Online]. Available:

https://doi.org/10.1109/TNNLS.2018.2886017

D. M. Bamasoud, A. S. Al-Dossary, N. M. Al-Harthy, R. A.

Al-Shomrany, G. S. Alghamdi, and R. O. Algahmdi, “Privacy and

security issues in cloud computing: A survey paper,” in International

Conference on Information Technology, ICIT 2021, Amman, Jordan, July

-15, 2021. IEEE, 2021, pp. 387–392. [Online]. Available: https:

//doi.org/10.1109/ICIT52682.2021.9491632

Y. Zhang and R. Sion, “Speculative execution attacks and cloud

security,” in Proceedings of the 2019 ACM SIGSAC Conference on

Cloud Computing Security Workshop, CCSW@CCS 2019, London, UK,

November 11, 2019, R. Sion and C. Papamanthou, Eds. ACM, 2019,

p. 201. [Online]. Available: https://doi.org/10.1145/3338466.3360287

Y. Alghofaili, A. Albattah, N. Alrajeh, M. A. Rassam, and B. A. S.

Al-rimy, “Secure cloud infrastructure: A survey on issues, current

solutions, and open challenges,” Applied Sciences, vol. 11, no. 19, 2021.

[Online]. Available: https://www.mdpi.com/2076-3417/11/19/9005

N. Dowlin, R. Gilad-Bachrach, K. Laine, K. Lauter, M. Naehrig, and

J. Wernsing, “Cryptonets: Applying neural networks to encrypted data

with high throughput and accuracy,” Tech. Rep. MSR-TR-2016-3,

[Online]. Available: https://www.microsoft.com/en-us/research/

publication/cryptonets-applying-neural-networks-to-encrypted-data-

with-high-throughput-and-accuracy/

J. Alvarez-Valle, P. Bhatu, N. Chandran, D. Gupta, A. Nori, A. Rastogi,

M. Rathee, R. Sharma, and S. Ugare, “Secure medical image analysis

with cryptflow,” 2020.

A. Soin, P. Bhatu, R. Takhar, N. Chandran, D. Gupta, J. Alvarez-Valle,

R. Sharma, V. Mahajan, and M. P. Lungren, “Multi-institution encrypted

medical imaging ai validation without data sharing,” 2021.

M. H. M. Elham Tabassi (NIST), Kevin Burns (MITRE). A taxonomy

and terminology of adversarial machine learning. [Online]. Available:

https://csrc.nist.gov/publications/detail/nistir/8269/draft

Y. LeCun and C. Cortes, “MNIST handwritten digit database,” 2010.

[Online]. Available: http://yann.lecun.com/exdb/mnist/

F. Boemer, A. Costache, R. Cammarota, and C. Wierzynski, “ngraph-

he2: A high-throughput framework for neural network inference on

encrypted data,” 2019.

F. Boemer, Y. Lao, R. Cammarota, and C. Wierzynski, “ngraph-he: A

graph compiler for deep learning on homomorphically encrypted data,”

A. Benaissa, B. Retiat, B. Cebere, and A. E. Belfedhal, “Tenseal: A

library for encrypted tensor operations using homomorphic encryption,”

S. Carpov, P. Dubrulle, and R. Sirdey, “Armadillo: A compilation

chain for privacy preserving applications,” in Proceedings of the 3rd

International Workshop on Security in Cloud Computing, ser. SCC ’15.

Association for Computing Machinery, 2015, p. 13–19. [Online].

Available: https://doi.org/10.1145/2732516.2732520

I. Chillotti, N. Gama, M. Georgieva, and M. Izabach`ene, “Faster fully

homomorphic encryption: Bootstrapping in less than 0.1 seconds,” Cryp-

tology ePrint Archive, Report 2016/870, 2016, https://ia.cr/2016/870.

S. S. Magara, C. Yildirim, F. Yaman, B. Dilekoglu, F. R. Tutas,

E. ̈Ozt ̈urk, K. Kaya, ̈O. Tastan, and E. Savas, “Ml with he: Privacy

preserving machine learning inferences for genome studies,” 2021.

R. Dathathri, O. Saarikivi, H. Chen, K. Laine, K. Lauter, S. Maleki,

M. Musuvathi, and T. Mytkowicz, “Chet: an optimizing compiler for

fully-homomorphic neural-network inferencing,” in Proceedings of the

th ACM SIGPLAN Conference on Programming Language Design and

Implementation, 2019, pp. 142–156.

E. Hesamifard, H. Takabi, M. Ghasemi, and R. N. Wright, “Privacy-

preserving machine learning as a service,” Proc. Priv. Enhancing

Technol., vol. 2018, no. 3, pp. 123–142, 2018. [Online]. Available:

https://doi.org/10.1515/popets-2018-0024

C. Boura, N. Gama, M. Georgieva, and D. Jetchev, “Chimera: Combin-

ing ring-lwe-based fully homomorphic encryption schemes,” Cryptology

ePrint Archive, Report 2018/758, 2018, https://ia.cr/2018/758.

Q. Lou, B. Feng, G. C. Fox, and L. Jiang, “Glyph: Fast and accurately

training deep neural networks on encrypted data,” 2020.

OpenMined. (2021) Tenseal library. [Online]. Available: https:

//github.com/OpenMined/TenSEAL

J. H. Cheon, D. Kim, D. Kim, H. H. Lee, and K. Lee, “Numerical

method for comparison on homomorphically encrypted numbers,” Cryp-

tology ePrint Archive, Report 2019/417, 2019, https://ia.cr/2019/417.

J. H. Cheon, D. Kim, and D. Kim, “Efficient homomorphic comparison

methods with optimal complexity,” Cryptology ePrint Archive, Report

/1234, 2019, https://ia.cr/2019/1234.

U. Michelucci, Advanced applied deep learning : convolutional neural

networks and object detection. Apress, 2019.

A. Dalskov, D. Escudero, and M. Keller, “Secure evaluation of quantized

neural networks,” Cryptology ePrint Archive, Report 2019/131, 2019,

https://ia.cr/2019/131.

OpenMined. (2021) Sympc library. [Online]. Available: https://

github.com/OpenMined/SyMPC

N. Kumar, M. Rathee, N. Chandran, D. Gupta, A. Rastogi, and

R. Sharma, “Cryptflow: Secure tensorflow inference,” 2020.

EXAMPLE OF ARTICLE FOR INTERNATIONAL JOURNALS OF ELECTRONICS AND TELECOMMUNICATIONS 7

D. Rathee, M. Rathee, N. Kumar, N. Chandran, D. Gupta, A. Rastogi,

and R. Sharma, “Cryptflow2: Practical 2-party secure inference,”

Proceedings of the 2020 ACM SIGSAC Conference on Computer and

Communications Security, 2020. [Online]. Available: http://dx.doi.org/

1145/3372297.3417274

D. Rathee, M. Rathee, R. K. K. Goli, D. Gupta, R. Sharma, N. Chandran,

and A. Rastogi, “Sirnn: A math library for secure rnn inference,” Cryp-

tology ePrint Archive, Report 2021/459, 2021, https://ia.cr/2021/459.

B. Knott, S. Venkataraman, A. Hannun, S. Sengupta, M. Ibrahim, and

L. van der Maaten, “Crypten: Secure multi-party computation meets

machine learning,” in arXiv 2109.00984, 2021.

M. Dahl, J. Mancuso, Y. Dupis, B. Decoste, M. Giraud, I. Livingstone,

J. Patriquin, and G. Uhma, “Private machine learning in tensorflow using

secure computation,” 2018.

W. Henecka, S. K ̈ogl, A.-R. Sadeghi, T. Schneider, and I. Wehrenberg,

“Tasty: Tool for automating secure two-party computations,” Cryptology

ePrint Archive, Report 2010/365, 2010, https://ia.cr/2010/365.

P. Mohassel and P. Rindal, “Aby¡sup¿3¡/sup¿: A mixed protocol

framework for machine learning,” in Proceedings of the 2018 ACM

SIGSAC Conference on Computer and Communications Security, ser.

CCS ’18. Association for Computing Machinery, 2018, p. 35–52.

[Online]. Available: https://doi.org/10.1145/3243734.3243760

S. Wagh, D. Gupta, and N. Chandran, “Securenn: Efficient and private

neural network training,” Cryptology ePrint Archive, Report 2018/442,

, https://ia.cr/2018/442.

W. Zheng, R. Deng, W. Chen, R. A. Popa, A. Panda, and I. Stoica,

“Cerebro: A platform for Multi-Party cryptographic collaborative

learning,” in 30th USENIX Security Symposium (USENIX Security 21).

USENIX Association, 2021, pp. 2723–2740. [Online]. Available:

https://www.usenix.org/conference/usenixsecurity21/presentation/zheng

S. Wagh, S. Tople, F. Benhamouda, E. Kushilevitz, P. Mittal, and

T. Rabin, “Falcon: Honest-majority maliciously secure framework for

private deep learning,” 2020.

M. S. Riazi, M. Samragh, H. Chen, K. Laine, K. Lauter, and F. Koushan-

far, “Xonn: Xnor-based oblivious deep neural network inference,” 2019.

M. S. Riazi, C. Weinert, O. Tkachenko, E. M. Songhori, T. Schneider,

and F. Koushanfar, “Chameleon: A hybrid secure computation frame-

work for machine learning applications,” 2018.

A.-R. Sadeghi and T. Schneider, “Generalized universal circuits for

secure evaluation of private functions with application to data clas-

sification,” Cryptology ePrint Archive, Report 2008/453, 2008, https:

//ia.cr/2008/453.

M. Barni, P. Failla, R. Lazzeretti, A.-R. Sadeghi, and T. Schneider,

“Privacy-preserving ecg classification with branching programs and

neural networks,” IEEE Transactions on Information Forensics and

Security, vol. 6, no. 2, pp. 452–468, 2011.

P. Mohassel and Y. Zhang, “Secureml: A system for scalable privacy-

preserving machine learning,” in 2017 IEEE Symposium on Security and

Privacy (SP), 2017, pp. 19–38.

N. Koti, A. Patra, R. Rachuri, and A. Suresh, “Tetrad: Actively secure

pc for secure training and inference,” Cryptology ePrint Archive,

Report 2021/755, 2021, https://ia.cr/2021/755.

A. Patra and A. Suresh, “Blaze: Blazing fast privacy-preserving machine

learning,” Proceedings 2020 Network and Distributed System Security

Symposium, 2020. [Online]. Available: http://dx.doi.org/10.14722/

ndss.2020.24202

N. Koti, M. Pancholi, A. Patra, and A. Suresh, “Swift: Super-fast and

robust privacy-preserving machine learning,” 2021.

EzPC. (2021) Ezpc. [Online]. Available: https://github.com/mpc-msri/

EzPC

PySyft. (2021) Pysyft. [Online]. Available: https://github.com/

OpenMined/PySyft

T. Ryffel, P. Tholoniat, D. Pointcheval, and F. Bach, “Ariann: Low-

interaction privacy-preserving deep learning via function secret sharing,”

D. Labs. (2021) tf-encrypted library. [Online]. Available: https:

//github.com/tf-encrypted/tf-encrypted

gramine. (2021) gramine, library. [Online]. Available: https:

//github.com/gramineproject/gramine

D. Labs. (2021) tf-trusted, library. [Online]. Available: https://

github.com/capeprivacy/tf-trusted

F. Tram`er and D. Boneh, “Slalom: Fast, verifiable and private execution

of neural networks in trusted hardware,” 2019.

F. Mo, H. Haddadi, K. Katevas, E. Marin, D. Perino, and N. Kourtel-

lis, “Ppfl: Privacy-preserving federated learning with trusted execution

environments,” 2021.

F. Mo, A. S. Shamsabadi, K. Katevas, S. Demetriou, I. Leontiadis,

A. Cavallaro, and H. Haddadi, “Darknetz,” Proceedings of the 18th

International Conference on Mobile Systems, Applications, and Services,

[Online]. Available: http://dx.doi.org/10.1145/3386901.3388946

J. J. Dai, Y. Wang, X. Qiu, D. Ding, Y. Zhang, Y. Wang, X. Jia,

L. C. Zhang, Y. Wan, Z. Li, J. Wang, S. Huang, Z. Wu, Y. Wang,

Y. Yang, B. She, D. Shi, Q. Lu, K. Huang, and G. Song, “Bigdl: A

distributed deep learning framework for big data,” in Proceedings of

the ACM Symposium on Cloud Computing, ser. SoCC’19. Association

for Computing Machinery, 2019, pp. 50–60. [Online]. Available:

https://arxiv.org/pdf/1804.05839.pdf

M. Orenbach, P. Lifshits, M. Minkin, and M. Silberstein, “Eleos: Exitless

os services for sgx enclaves,” in Proceedings of the Twelfth European

Conference on Computer Systems, ser. EuroSys ’17. Association

for Computing Machinery, 2017, p. 238–253. [Online]. Available:

https://doi.org/10.1145/3064176.3064219

R. Kunkel, D. L. Quoc, F. Gregor, S. Arnautov, P. Bhatotia, and

C. Fetzer, “Tensorscone: A secure tensorflow framework using intel sgx,”

W. Ozga, D. L. Quoc, and C. Fetzer, “Perun: Secure multi-stakeholder

machine learning framework with gpu support,” 2021.

A. Mondal, Y. More, R. H. Rooparaghunath, and D. Gupta, “Flatee:

Federated learning across trusted execution environments,” 2021.

LeNET. (2021) Lenet. [Online]. Available: https://en.wikipedia.org/

wiki/LeNet

Downloads

Published

2024-04-19

Issue

Section

Cryptography and Cybersecurity