Publication details

F. Galli, C. Palamidessi, T. Cucinotta. "Online Sensitivity Optimization in Differentially Private Learning," (to appear in) in Proceedings of the 38th Annual AAAI Conference on Artificial Intelligence, February 20-27th, 2024, Vancouver, Canada.

Abstract

Training differentially private machine learning models requires constraining an individual's contribution to the optimization process. This is achieved by clipping the 2-norm of their gradient at a predetermined threshold prior to averaging and batch sanitization. This selection adversely influences optimization in two opposing ways: it either exacerbates the bias due to excessive clipping at lower values, or augments sanitization noise at higher values. The choice significantly hinges on factors such as the dataset, model architecture, and even varies within the same optimization, demanding meticulous tuning usually accomplished through a grid search. In order to circumvent the privacy expenses incurred in hyper-parameter tuning, we present a novel approach to dynamically optimize the clipping threshold. We treat this threshold as an additional learnable parameter, establishing a clean relationship between the threshold and the cost function. This allows us to optimize the former with gradient descent, with minimal repercussions on the overall privacy analysis. Our method is thoroughly assessed against alternative fixed and adaptive strategies across diverse datasets, tasks, model dimensions, and privacy levels. Our results indicate that it performs comparably or better in the evaluated scenarios, given the same privacy requirements.

See paper on publisher's website

Download paper

DOI: 10.1609/aaai.v38i11.29099

BibTeX entry:

@article{Galli_2024,
     title={Online Sensitivity Optimization in Differentially Private Learning},
     volume={38},
     ISSN={2159-5399},
     url={http://dx.doi.org/10.1609/aaai.v38i11.29099},
     DOI={10.1609/aaai.v38i11.29099},
     number={11},
     journal={Proceedings of the AAAI Conference on Artificial Intelligence},
     publisher={Association for the Advancement of Artificial Intelligence (AAAI)},
     author={Galli, Filippo and Palamidessi, Catuscia and Cucinotta, Tommaso},
     year={2024},
     month=mar,
     pages={12109–12117}
}

Last updated on
18 March 2024