scholar_flux.security package

Submodules

scholar_flux.security.filters module

The scholar_flux.security.filters module implements the basic foundational filter used by logging utilities to determine how text will be masked and redacted prior to being stored in file handlers or written to the console.

This module uses the SensitiveDataMasker class to flag and redact text prior to it ever reaching the console or file via the logging module.

class scholar_flux.security.filters.MaskingFilter(masker: SensitiveDataMasker | None = None)

Bases: Filter

Custom class for adding a masking to the logs: Uses the SensitiveDataMasker in order to enforce the rules detailing which fields should be logged. By default, the SensitiveDataMasker masks API keys and email parameters in requests to APIs sent via scholar_flux.

Initialized MaskingFilters can also be updated without re-instantiation by adding or removing patterns from the set of all patterns within the received Masker.

This class can otherwise be added to other loggers with minimal effort::

>>> import logging
>>> from scholar_flux.security import MaskingFilter # contains the filter
>>> formatting = "%(name)s - %(levelname)s - %(message)s" # custom format
>>> logger = logging.getLogger('security_logger') # gets or creates a new logger
>>> logging.basicConfig(level=logging.DEBUG, format=formatting) # set the level and formatting for the log
>>> logging_filter = MaskingFilter() # creating a new filter
>>> logger.addFilter(logging_filter) # adding the filter and formatting rules
>>> logger.info("The following api key should be filtered: API_KEY='an_api_key_that_needs_to_be_filtered'")
# OUTPUT: security_logger - INFO - The following api key should be filtered: API_KEY='***'

__init__(masker: SensitiveDataMasker | None = None)

By default, this implementation is applied in the initialization of the package in scholar_flux.__init__ on import, so this class does not need to be applied directly.

Parameters:

masker (Optional[SensitiveDataMasker]) – The actual implementation responsible for masking text matching patterns. Note that if a masker is not passed, the MaskingFilter will initialize a masker directly.

filter(record: LogRecord) → bool

Helper method used by the logging.Logger class when adding custom filters to the logging module.

Masks string, dictionary, and list data in log records.

This class will always return True after an attempt to mask sensitive fields is completed.

scholar_flux.security.masker module

The scholar_flux.security.masker defines the SensitiveDataMasker that is used during API retrieval and processing.

The SensitiveDataMasker implements the logic necessary to determine text strings to mask and is used to identify and mask potentially sensitive fields based on dictionary fields and string-based patterns.

This class is also used during initialization and within the scholar_flux.SearchAPI class to identify and mask API keys, emails, and other forms of sensitive data with the aim of redacting text from both console and file system logs.

class scholar_flux.security.masker.SensitiveDataMasker(register_defaults: bool = True)

Bases: object

The main interface used by the scholar_flux API for masking all text identified as sensitive.

This class is used by scholar_flux to ensure that all sensitive text sent to the scholar_flux.logger is masked.

The SensitiveDataMasker operates through the registration of patterns that identify the text to mask.

Components:

• KeyMaskingPattern:

  identifies specific keys and regex patterns that will signal text to filter

• StringMaskingPattern:

  identifies strings to filter either by fixed or pattern matching

• MaskingPatternSet:

  A customized set accepting only subclasses of MaskingPatterns that specify the rules for filtering text of sensitive fields.

By default, this structure implements masking for email addresses, API keys, bearer tokens, etc. that are identified as sensitive parameters/secrets.

Parameters:

register_defaults (bool) – Determines whether or not to add the patterns that filter API keys email parameters and auth bearers.

Examples

>>> from scholar_flux.security import SensitiveDataMasker # imports the class
>>> masker = SensitiveDataMasker(register_defaults = True) # initializes a masker with defaults
>>> masked = masker.mask_text("'API_KEY' = 'This_Should_Be_Masked_1234', email='a.secret.email@address.com'")
>>> print(masked)
# OUTPUT: 'API_KEY' = '***', email='***'

# specifying a new secret to filter: uses regex by default >>> new_secret = “This string should be filtered” >>> masker.add_sensitive_string_patterns(name=’custom’, patterns=new_secret, use_regex = False)

# applying the filter >>> masked = masker.mask_text(f”The following string should be masked: {new_secret}”) >>> print(masked) # OUTPUT: The following string should be masked: ***

__init__(register_defaults: bool = True) → None

Initializes the SensitiveDataMasker for registering and applying different masking patterns.

Each registered pattern defines a fixed or regular expression pattern to be scrubbed from text with via SensitiveDataMasker.mask_text when matched.

Parameters:

• register_defaults (bool) – Indicates whether to register_defaults for scrubbing emails,

• api_keys

• Bearers (Authorization)

• self.mask_text (etc. from the text when applying)

self.patterns

Indicates the full list of patterns that will be applied when scrubbing text of sensitive fields using masking patterns.

Type:

Set[MaskingPattern]

add_pattern(pattern: MaskingPattern) → None

Adds a pattern to the self.patterns attribute.

add_sensitive_key_patterns(name: str, fields: List[str] | str, fuzzy: bool = False, **kwargs: Any) → None

Adds patterns that identify potentially sensitive strings with the aim of filtering them from logs.

The parameters provided to the method are used to create new string patterns.

Parameters:

• name (str) – The name associated with the pattern (aids identification of patterns)

• fields (List[str] | str) – The list of fields to identify to search and remove from logs.

• pattern (str) – An optional parameter for filtering and removing sensitive fields that match a given pattern. By default this is already set to remove api keys that are typically denoted by alpha numeric fields

• fuzzy (bool) – If true, regular expressions are used to identify keys. Otherwise the fixed (field) key matching is used through the implementation of a basic KeyMaskingPattern.

• **kwargs – Other fields, specifiable via additional keyword arguments that are passed to KeyMaskingPattern

add_sensitive_string_patterns(name: str, patterns: List[str] | str, **kwargs: Any) → None

Adds patterns that identify potentially sensitive strings with the aim of filtering them from logs.

The parameters provided to the method are used to create new string patterns.

Parameters:

• name (str) – The name associated with the pattern (aids identification of patterns)

• patterns (List[str] | str) – The list of patterns to search for and remove from logs

• **kwargs – Other fields, specifiable via additional keyword arguments used to create the StringMaskingPattern

clear() → None

Clears the SensitiveDataMasker.patterns set of all previously registered MaskingPatterns.

This method also clears patterns that were registered by default on the initialization of the SensitiveDataMasker.

The masker would otherwise use the available patterns set to determine what text strings would be masked when the mask_text method is called. Calling mask_text after clearing all MaskingPatterns from the current masker will leave all text unmasked and return the inputted text as is.

get_patterns_by_name(name: str) → Set[MaskingPattern]

Retrieves all patterns with names matching the provided name.

classmethod is_secret(obj: Any) → bool

Utility method for verifying whether the current value is a secret.

This method delegates the verification of the value type to the SecretUtils helper class to abstract the implementation details in cases where the implementation details might require modification in the future for special cases.

Parameters:

obj (Any) – The object to check

Returns:

True if the object is a SecretStr, False otherwise

Return type:

bool

mask_dict(data: dict, convert_objects: bool = False) → dict

Masks sensitive values in dictionaries based on registered key patterns.

This method provides more reliable masking for structured data than string pattern matching, as it directly matches dictionary keys rather than parsing formatted text.

Parameters:

• data (dict) – The dictionary to mask

• convert_objects (bool) – Defines whether to convert data objects (BaseModels, dataclasses) as masked strings

Returns:

New dictionary with sensitive values masked

Return type:

(dict)

Example

>>> masker = SensitiveDataMasker()
>>> config = {'password': 'secret123', 'host': 'localhost'}
>>> masked = masker.mask_dict(config)
>>> print(masked)
# OUTPUT: {'password': '***', 'host': 'localhost'}

mask_list(data: list | tuple, convert_objects: bool = False) → list

Masks sensitive values in lists based on registered patterns.

Recursively processes nested structures (dicts, lists, tuples) and applies masking patterns to any sensitive content found.

Parameters:

data (list | tuple) – list or tuple to mask

Returns:

New list with sensitive values masked

Return type:

list

Note

For type-safety, tuples are converted into lists and may need to be converted as a tuple if used as input.

Example

>>> masker = SensitiveDataMasker()
>>> configs = [{'password': 'secret'}, 'api_key=12345']
>>> masked = masker.mask_list(configs)
>>> print(masked)
# OUTPUT: [{'password': '***'}, 'api_key=***']

mask_object(value: Any) → Any

Converts objects into their masked string representation.

Parameters:

value (Any) – The value to convert and mask

Returns:

Masked string representation of the value

Return type:

str

mask_output(convert_objects: bool = False) → Callable[[Callable[[P], R]], Callable[[P], R]]

Decorator that wraps a function or method using the current SensitiveDataMasker to mask sensitive values.

Parameters:

convert_objects (bool) – If True, objects of unknown types are converted to strings before masking. Otherwise, no conversion takes place, and only strings (passed explicitly or nested within dictionaries, lists, sets) are masked. Defaults to False.

Returns:

A decorator that wraps a function, masking its output according to the registered patterns.

Return type:

Callable[[Callable[P, R]], Callable[P, R]]

Note

The decorated function’s signature and docstring are preserved via functools.wraps.

Example

>>> import os
>>> from scholar_flux import masker
>>> @masker.mask_output()
... def default_config():
...     return {'API_KEY': os.environ.get('EXAMPLE_API_KEY'), 'host': 'https://example-api.com'}
>>> default_config()
# OUTPUT: {'API_KEY': '***', 'host': 'https://example-api.com'}

static mask_secret(obj: None) → None

static mask_secret(obj: Any) → SecretStr

Method for ensuring that any non-secret keys will be masked as secrets.

Parameters:

obj (Any) – An object to attempt to unmask if it is a secret string

Returns:

A SecretStr representation of the original object

Return type:

obj (SecretStr)

mask_text(text: str) → str

Public method for removing sensitive data from text/logs.

Note that the data that is redacted is dependent on what patterns were already previously defined in the SensitiveDataMasker. By default, this includes API keys, emails, and auth headers.

Parameters:

text (str) – the text to scrub of sensitive data

Returns:

The cleaned text that excludes sensitive fields

Return type:

str

mask_value(value: Any, convert_objects: bool = False) → Any

Recursively masks a single value based on its type.

register_secret_if_exists(field: str, value: SecretStr | Any, name: str | None = None, use_regex: bool = False, ignore_case: bool = True) → bool

Identifies fields already registered as secret strings and adds a relevant pattern for ensuring that the field, when unmasked for later use, doesn’t display in logs. Note that if the current field is not a SecretStr, the method will return False without modification or side-effects.

The parameters provided to the method are used to create new string patterns when a SecretStr is detected.

Parameters:

• field (str) – The field, parameter, or key associated with the secret key

• value (SecretStr | Any) – The value, if typed as a secret string, to be registered as a pattern

• name (Optional[str]) – The name to add to identify the relevant pattern by within the pattern set. If not provided, defaults to the field name.

• use_regex (bool) – Indicates whether the current function should use regular expressions when matching the pattern in text. Defaults to False.

• ignore_case (bool) – Whether we should consider case when determining whether or not to filter a string. Defaults to True.

Returns:

If the value is a SecretStr, a string masking pattern is registered for the value and True is returned. if the value is not a SecretStr, False is returned and no side-effects will occur in this case.

Return type:

bool

Example

>>> masker = SensitiveDataMasker()
>>> api_key = SecretStr("sk-123456")
>>> registered = masker.register_secret_if_exists("api_key", api_key)
>>> print(registered)
# OUTPUT: True
>>> registered = masker.register_secret_if_exists("normal_field", "normal_value")
>>> print(registered)
# OUTPUT: False

remove_pattern_by_name(name: str) → int

Removes patterns by name and returns the total number of removed patterns.

structure(flatten: bool = False, show_value_attributes: bool = False) → str

Displays a representation of the current SensitiveDataMasker in a human-readable format.

Parameters:

• flatten (bool) – Indicates whether the SensitiveDataMasker representation should be displayed on a single line.

• show_value_attributes (bool) – Indicates whether the masking patterns should be shown in the console (False by default).

Returns:

A structural representation of the current SensitiveDataMasker

Return type:

str

static unmask_secret(obj: Any) → Any

Method for ensuring that usable values can be successfully extracted from objects.

If the current value is a secret string, this method will return the secret value from the object.

Parameters:

obj (Any) – An object to attempt to unmask if it is a secret string

Returns:

The object’s original type before being converted into a secret string

Return type:

obj (Any)

update(pattern: MaskingPattern | Set[MaskingPattern] | Set[KeyMaskingPattern] | Set[StringMaskingPattern] | MutableSequence[MaskingPattern | KeyMaskingPattern | StringMaskingPattern]) → None

Adds a pattern to the self.patterns attribute.

scholar_flux.security.patterns module

The scholar_flux.security.patterns module implements pattern matching utilities for sensitive string masking.

This module defines the foundational patterns required to implement lightweight fixed/regular expression pattern matching objects for masking sensitive strings in both text and JSON-formatted dictionaries.

Classes:

MaskingPattern:

Implements the abstract base class that defines how MaskingPatterns are created and formatted.

MaskingPatternSet:

Defines a subclass of a set that accepts only subclasses of MaskingPatterns for robustness.

KeyMaskingPattern:

Defines the class and methods necessary to mask text based on the presence or absence of a specific field name when determining which patterns to mask.

StringMaskingPattern:

Defines the class and methods necessary to mask text based on the presence or absence of specific patterns. These patterns can either be fixed or regular expressions and accept both case-sensitive and case-insensitive pattern matching settings.

class scholar_flux.security.patterns.FuzzyKeyMaskingPattern(name: str, pattern: str | SecretStr = '[A-Za-z0-9\\-_]+', apply_to_dict: bool = True, field: str = <factory>, replacement: str = '***', use_regex: bool = True, ignore_case: bool = True, mask_pattern: bool = True, *, apply_to_text: bool = True)

Bases: KeyMaskingPattern

A KeyMaskingPattern subclass that allows the field parameter to use regular expressions field pattern matching.

name

The name to be associated with a particular pattern - can help in later identification and retrieval of rules associated with pattern masks of a particular category.

Type:

str

field

The regular expression field to look for when determining whether to mask a specific parameter.

Type:

str

pattern

The pattern to use to remove sensitive fields, contingent on a parameter being defined. By default, the pattern is set to allow for the removal dashes and alphanumeric fields but can be overridden based on API specific specifications.

Type:

str

replacement

Indicates the replacement string for the value in the key-value pair if matched (’***’ by default)

Type:

str

use_regex

Indicates whether the current function should use regular expressions

Type:

bool

ignore_case

whether we should consider case when determining whether or not to filter a string. (True by default)

Type:

bool

mask_pattern

Indicates whether we should, by default, mask pattern strings that are registered in the MaskingPattern. This is True by default.

Type:

bool

apply_to_text

Whether this pattern should be applied during mask_text() operations. Defaults to True.

Type:

bool

apply_to_dict

Whether this pattern should be applied during mask_dict() operations. Defaults to True.

Type:

bool

apply_masking(text: str) → str

Uses fuzzy field matching to identify fields containing sensitive data in text.

This method is revised to account for circumstances where several fields might be present in the same text string using the | delimiter. The masker can be customized using the following fields: field, pattern, replacement, use_regex, and ignore_case.

Parameters:

text (str) – The text to clean of sensitive fields

matches_key(key: str) → bool

Checks if a dictionary key matches this pattern’s field using regular expressions.

This method returns True if the field of the current FuzzyKeyMaskingPattern, when used as a regular expression, matches the provided key and false otherwise.

Parameters:

key (str) – The dictionary key to check

Returns:

True if the key matches the regex pattern, False otherwise

Return type:

bool

Example

>>> pattern = FuzzyKeyMaskingPattern(name="test", field="pass(word)?")
>>> pattern.matches_key("password")
# OUTPUT: True
>>> pattern.matches_key("pass")
# OUTPUT: True
>>> pattern.matches_key("Pass") # (ignore_case=True is the default)
# OUTPUT: True
>>> pattern.matches_key("passphrase") # (regex matching)
# OUTPUT: True

class scholar_flux.security.patterns.KeyMaskingPattern(name: str, pattern: str | SecretStr = '[A-Za-z0-9\\-_]+', apply_to_dict: bool = True, field: str = <factory>, replacement: str = '***', use_regex: bool = True, ignore_case: bool = True, mask_pattern: bool = True, *, apply_to_text: bool = True)

Bases: MaskingPattern

Masks values associated with specific keys/fields/parameters in text and API requests.

The KeyMaskingPattern identifies fields in dumped JSON-formatted data that are commonly prepared in the creation of request URLs. After identifying the assigned fixed-string field or key in a request URL or string-formatted dictionary, the pattern conditionally masks its associated value using a fixed or regular expression pattern.

By default, the masking pattern is set to filter string combinations of dashes and alphanumeric fields that are commonly observed in API keys, secrets, etc. The pattern parameter can be overridden to identify sensitive text such as birthdays, combinations of digits, and addresses using regular expressions.

name

The name to be associated with a particular pattern. Facilitates the identification and retrieval of pattern masks by name/category in later steps.

Type:

str

field

The fixed field string to look for when determining whether to mask a specific parameter.

Type:

str

pattern

The pattern that will be used to identify and mask sensitive fields when its corresponding field/JSON key has been located.

Type:

str

replacement

Indicates the replacement string for the value in the key-value pair if matched (’***’ by default)

Type:

str

use_regex

Indicates whether the current function should use regular expressions

Type:

bool

ignore_case

whether we should consider case when determining whether or not to filter a string. (True by default)

Type:

bool

mask_pattern

Indicates whether we should, by default, mask pattern strings that are registered in the MaskingPattern. This is True by default.

Type:

bool

apply_to_text

Whether this pattern should be applied during mask_text() operations. Defaults to True. Set to False for dict-only patterns that would cause false positives in text.

Type:

bool

apply_to_dict

Whether this pattern should be applied during mask_dict() operations. Defaults to True. Set to False for text-only patterns (like URL patterns) that don’t apply to dictionary keys.

Type:

bool

__init__(*args: Any, **kwargs: Any) → None

apply_masking(text: str) → str

Removes sensitive fields from text based on the current KeyMaskingPattern’s defined settings.

For the current pattern to trigger, the current field must match the string field of a dictionary or the name of the field found in the current text string. Masking behavior can be customized based on the defined settings of the current masking pattern (field, pattern, replacement, and ignore_case)

Parameters:

text (str) – The text to clean of sensitive fields

apply_to_dict: bool = True

field: str = FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1)])

ignore_case: bool = True

mask_pattern: bool = True

matches_key(key: str) → bool

Checks if a dictionary key matches this pattern’s field.

This method returns True when the field of the current KeyMaskingPattern matches the key. If the field doesn’t match, False is returned. A case insensitive match is only performed if KeyMaskingPattern.ignore_case=True.

Parameters:

key (str) – The dictionary key to check

Returns:

True if the key matches, False otherwise

Return type:

bool

Example

>>> pattern = KeyMaskingPattern(name="test", field="password")
>>> pattern.matches_key("password")
# OUTPUT: True
>>> pattern.matches_key("PASSWORD") # (ignore_case=True by default)
# OUTPUT: True
>>> pattern.matches_key("pass") # (exact match required)
# OUTPUT: False

name: str

pattern: str | SecretStr = '[A-Za-z0-9\\-_]+'

replacement: str = '***'

use_regex: bool = True

class scholar_flux.security.patterns.MaskingPattern(name: str, pattern: str | SecretStr, *, apply_to_text: bool = True, apply_to_dict: bool = False)

Bases: ABC

The base class for creating MaskingPattern objects that can be used to mask fields based on defined rules.

__init__(*args: Any, **kwargs: Any) → None

abstract apply_masking(text: str) → str

Base method that will be overridden by subclasses to later remove sensitive values and fields from text.

apply_to_dict: bool = False

apply_to_text: bool = True

name: str

pattern: str | SecretStr

class scholar_flux.security.patterns.MaskingPatternSet

Bases: set[MaskingPattern]

Subclasses the set to implement the type-safe storage of MaskingPatterns.

As a result, the robustness of operations requiring sensitive pattern masking is increased, reducing the likelihood of unexpected errors during response retrieval and the redaction of sensitive parameters with ScholarFlux.

__init__() → None

Initializes the MaskingPatternSet as an empty set.

add(item: MaskingPattern) → None

Overrides the basic add method to ensure that each item is type-checked prior to entering the set.

update(*others: Iterable[MaskingPattern]) → None

Overrides the basic update method to ensure that all items are type-checked prior to entering the set.

class scholar_flux.security.patterns.StringMaskingPattern(name: str, pattern: str | SecretStr, replacement: str = '***', use_regex: bool = True, ignore_case: bool = True, mask_pattern: bool = True, *, apply_to_text: bool = True, apply_to_dict: bool = False)

Bases: MaskingPattern

Masks values associated with a particular pattern or fixed string in text and API requests.

name

The name to be associated with a particular pattern - can help in later identification and retrieval of rules associated with pattern masks of a particular category.

Type:

str

pattern

The pattern to use to remove sensitive fields, contingent on a parameter being defined. By default, the pattern is set to allow for the removal dashes and alphanumeric fields but can be overridden based on API specific specifications.

Type:

str

replacement

Indicates the replacement string for the value in the string if matched (’***’ by default)

Type:

str

use_regex

Indicates whether the current function should use regular expressions

Type:

bool

ignore_case

whether we should consider case when determining whether or not to filter a string. (True by default)

Type:

bool

mask_pattern

Indicates whether we should, by default, mask pattern strings that are registered in the MaskingPattern. This is True by default.

Type:

bool

apply_to_text

Whether this pattern should be applied during mask_text() operations. Defaults to True.

Type:

bool

apply_to_dict

Whether this pattern should be applied to dictionary values during mask_dict() operations. Defaults to False. String patterns typically match text content (like URLs) rather than dict keys or values.

Type:

bool

Note

By default, this class does not operate on dictionaries unless explicitly configured to do so.

__init__(*args: Any, **kwargs: Any) → None

apply_masking(text: str) → str

Removes sensitive strings or patterns from text based on the defined settings of the StringMaskingPattern.

For the current pattern to trigger, the registered pattern must match the text exactly if use_regex=False. Otherwise, the pattern is treated as a regular expression. Case sensitivity is determined by the ignore_case attribute.

Masking behavior can be further customized based on the defined settings of the current masking pattern based on the attributes specified for pattern, replacement, use_regex, and ignore_case.

Parameters:

text (str) – The text to clean of sensitive fields

Returns:

The text after scrubbing sensitive fields

Return type:

text (str)

ignore_case: bool = True

mask_pattern: bool = True

name: str

pattern: str | SecretStr

replacement: str = '***'

use_regex: bool = True

scholar_flux.security.utils module

The scholar_flux.security.utils module defines the SecretUtils class that implements the basic set of tools for both masking and unmasking text, and identifying if a field is masked.

This class uses the pydantic.SecretStr class to mask and unmask fields and can be further extended to encrypt and decrypt text as needed before and after conversion to a secret string, respectively.

class scholar_flux.security.utils.SecretUtils

Bases: object

Helper utility for both masking and unmasking strings.

Class methods are defined so that they can be used directly or implemented as a mixin so that subclasses can implement the class methods directly.

classmethod is_secret(obj: Any) → bool

Utility class method used to verify whether the current variable is a secret string. This method abstracts the implementation details into a single method to aid further extensibility.

Parameters:

obj (Any) – The object to check

Returns:

True if the object is a SecretStr, False otherwise

Return type:

bool

classmethod mask_secret(obj: None) → None

classmethod mask_secret(obj: Any) → SecretStr

Helper method masking variables into secret strings:

Parameters:

obj (Any | SecretStr) – An object to attempt to unmask if it is a secret string

Returns:

A SecretStr representation of the original object

Return type:

obj (SecretStr)

Examples

>>> from scholar_flux.security import SecretUtils
>>> string = 'a secret'
>>> secret_string = SecretUtils.mask_secret(string)
>>> isinstance(secret_string, SecretStr) is True
# OUTPUT: True

>>> no_string = None
>>> non_secret = SecretUtils.mask_secret(no_string)
>>> non_secret is None
# OUTPUT: True

classmethod unmask_secret(obj: Any) → Any

Helper method for unmasking a variable from a SecretStr into its native type if a secret string.

Parameters:

obj (Any | SecretStr) – An object to attempt to unmask if it is a secret string

Returns:

The object’s original type before being converted into a secret string

Return type:

obj (Any)

Examples

>>> from scholar_flux.security import SecretUtils
>>> from pydantic import SecretStr
>>> string = 'a secret'
>>> secret_string = SecretUtils.mask_secret(string)
>>> isinstance(secret_string, SecretStr) is True
# OUTPUT: True
>>> SecretUtils.unmask_secret(secret_string) == string
# OUTPUT: True
>>> SecretUtils.unmask_secret(None) is None
# OUTPUT: True

Module contents

The scholar_flux.security module contains classes and models created specifically for ensuring that console and file logs do not contain sensitive data. The set of modules uses pattern matching to determine whether, when sending a request, any known API keys are filtered from the logs.

Core classes:

• SecretUtils: Class with basic static methods for masking and unmasking non-missing strings with pydantic.SecretStr

• MaskingPattern: Basic pattern from which all subclasses inherit from in order to define rules for masking strings

• KeyMaskingPattern: Matches key-value pairs for commonly sensitive fixed string fields (e.g. api_key, mailto)

• FuzzyKeyMaskingPattern: Extends the KeyMaskingPattern for fuzzy field matching when parameter names may vary

• StringMaskingPattern: Identifies and masks known sensitive strings using either regex or fixed string matching

• MaskingFilter: Defines the core logging filter used by the dedicated scholar_flux.logger to hide sensitive info

• MaskingPatternSet: Container that will hold a set of all String- and Key-based patterns used in the package

• SensitiveDataMasker: Main entry point for managing/adding to/deleting from the list of all patterns to be filtered

Note that the global package level SensitiveDataMasker is instantiated on package loading and can be imported:

>>> from scholar_flux import masker
>>> print(masker) # view all currently masked strings and keys
# OUTPUT: SensitiveDataMasker(patterns=MaskingPatternSet(...))

# Set up and remove all matching email-like strings >>> email_pattern = r”[a-zA-Z0-9._%+-]+(@|%40)[a-zA-Z0-9.-]+[.][a-zA-Z]+” >>> masker.add_sensitive_string_patterns( name=”email_strings”, patterns=email_pattern, use_regex = True) >>> masker.mask_text(”here_is_my_fake123@email.com”) # OUTPUT: ‘***’

class scholar_flux.security.FuzzyKeyMaskingPattern(name: str, pattern: str | SecretStr = '[A-Za-z0-9\\-_]+', apply_to_dict: bool = True, field: str = <factory>, replacement: str = '***', use_regex: bool = True, ignore_case: bool = True, mask_pattern: bool = True, *, apply_to_text: bool = True)

Bases: KeyMaskingPattern

A KeyMaskingPattern subclass that allows the field parameter to use regular expressions field pattern matching.

name

The name to be associated with a particular pattern - can help in later identification and retrieval of rules associated with pattern masks of a particular category.

Type:

str

field

The regular expression field to look for when determining whether to mask a specific parameter.

Type:

str

pattern

The pattern to use to remove sensitive fields, contingent on a parameter being defined. By default, the pattern is set to allow for the removal dashes and alphanumeric fields but can be overridden based on API specific specifications.

Type:

str

replacement

Indicates the replacement string for the value in the key-value pair if matched (’***’ by default)

Type:

str

use_regex

Indicates whether the current function should use regular expressions

Type:

bool

ignore_case

whether we should consider case when determining whether or not to filter a string. (True by default)

Type:

bool

mask_pattern

Indicates whether we should, by default, mask pattern strings that are registered in the MaskingPattern. This is True by default.

Type:

bool

apply_to_text

Whether this pattern should be applied during mask_text() operations. Defaults to True.

Type:

bool

apply_to_dict

Whether this pattern should be applied during mask_dict() operations. Defaults to True.

Type:

bool

apply_masking(text: str) → str

Uses fuzzy field matching to identify fields containing sensitive data in text.

This method is revised to account for circumstances where several fields might be present in the same text string using the | delimiter. The masker can be customized using the following fields: field, pattern, replacement, use_regex, and ignore_case.

Parameters:

text (str) – The text to clean of sensitive fields

matches_key(key: str) → bool

Checks if a dictionary key matches this pattern’s field using regular expressions.

This method returns True if the field of the current FuzzyKeyMaskingPattern, when used as a regular expression, matches the provided key and false otherwise.

Parameters:

key (str) – The dictionary key to check

Returns:

True if the key matches the regex pattern, False otherwise

Return type:

bool

Example

>>> pattern = FuzzyKeyMaskingPattern(name="test", field="pass(word)?")
>>> pattern.matches_key("password")
# OUTPUT: True
>>> pattern.matches_key("pass")
# OUTPUT: True
>>> pattern.matches_key("Pass") # (ignore_case=True is the default)
# OUTPUT: True
>>> pattern.matches_key("passphrase") # (regex matching)
# OUTPUT: True

class scholar_flux.security.KeyMaskingPattern(name: str, pattern: str | SecretStr = '[A-Za-z0-9\\-_]+', apply_to_dict: bool = True, field: str = <factory>, replacement: str = '***', use_regex: bool = True, ignore_case: bool = True, mask_pattern: bool = True, *, apply_to_text: bool = True)

Bases: MaskingPattern

Masks values associated with specific keys/fields/parameters in text and API requests.

The KeyMaskingPattern identifies fields in dumped JSON-formatted data that are commonly prepared in the creation of request URLs. After identifying the assigned fixed-string field or key in a request URL or string-formatted dictionary, the pattern conditionally masks its associated value using a fixed or regular expression pattern.

By default, the masking pattern is set to filter string combinations of dashes and alphanumeric fields that are commonly observed in API keys, secrets, etc. The pattern parameter can be overridden to identify sensitive text such as birthdays, combinations of digits, and addresses using regular expressions.

name

The name to be associated with a particular pattern. Facilitates the identification and retrieval of pattern masks by name/category in later steps.

Type:

str

field

The fixed field string to look for when determining whether to mask a specific parameter.

Type:

str

pattern

The pattern that will be used to identify and mask sensitive fields when its corresponding field/JSON key has been located.

Type:

str

replacement

Indicates the replacement string for the value in the key-value pair if matched (’***’ by default)

Type:

str

use_regex

Indicates whether the current function should use regular expressions

Type:

bool

ignore_case

whether we should consider case when determining whether or not to filter a string. (True by default)

Type:

bool

mask_pattern

Indicates whether we should, by default, mask pattern strings that are registered in the MaskingPattern. This is True by default.

Type:

bool

apply_to_text

Whether this pattern should be applied during mask_text() operations. Defaults to True. Set to False for dict-only patterns that would cause false positives in text.

Type:

bool

apply_to_dict

Whether this pattern should be applied during mask_dict() operations. Defaults to True. Set to False for text-only patterns (like URL patterns) that don’t apply to dictionary keys.

Type:

bool

__init__(*args: Any, **kwargs: Any) → None

apply_masking(text: str) → str

Removes sensitive fields from text based on the current KeyMaskingPattern’s defined settings.

For the current pattern to trigger, the current field must match the string field of a dictionary or the name of the field found in the current text string. Masking behavior can be customized based on the defined settings of the current masking pattern (field, pattern, replacement, and ignore_case)

Parameters:

text (str) – The text to clean of sensitive fields

apply_to_dict: bool = True

field: str = FieldInfo(annotation=NoneType, required=True, metadata=[MinLen(min_length=1)])

ignore_case: bool = True

mask_pattern: bool = True

matches_key(key: str) → bool

Checks if a dictionary key matches this pattern’s field.

This method returns True when the field of the current KeyMaskingPattern matches the key. If the field doesn’t match, False is returned. A case insensitive match is only performed if KeyMaskingPattern.ignore_case=True.

Parameters:

key (str) – The dictionary key to check

Returns:

True if the key matches, False otherwise

Return type:

bool

Example

>>> pattern = KeyMaskingPattern(name="test", field="password")
>>> pattern.matches_key("password")
# OUTPUT: True
>>> pattern.matches_key("PASSWORD") # (ignore_case=True by default)
# OUTPUT: True
>>> pattern.matches_key("pass") # (exact match required)
# OUTPUT: False

name: str

pattern: str | SecretStr = '[A-Za-z0-9\\-_]+'

replacement: str = '***'

use_regex: bool = True

class scholar_flux.security.MaskingFilter(masker: SensitiveDataMasker | None = None)

Bases: Filter

Custom class for adding a masking to the logs: Uses the SensitiveDataMasker in order to enforce the rules detailing which fields should be logged. By default, the SensitiveDataMasker masks API keys and email parameters in requests to APIs sent via scholar_flux.

Initialized MaskingFilters can also be updated without re-instantiation by adding or removing patterns from the set of all patterns within the received Masker.

This class can otherwise be added to other loggers with minimal effort::

>>> import logging
>>> from scholar_flux.security import MaskingFilter # contains the filter
>>> formatting = "%(name)s - %(levelname)s - %(message)s" # custom format
>>> logger = logging.getLogger('security_logger') # gets or creates a new logger
>>> logging.basicConfig(level=logging.DEBUG, format=formatting) # set the level and formatting for the log
>>> logging_filter = MaskingFilter() # creating a new filter
>>> logger.addFilter(logging_filter) # adding the filter and formatting rules
>>> logger.info("The following api key should be filtered: API_KEY='an_api_key_that_needs_to_be_filtered'")
# OUTPUT: security_logger - INFO - The following api key should be filtered: API_KEY='***'

__init__(masker: SensitiveDataMasker | None = None)

By default, this implementation is applied in the initialization of the package in scholar_flux.__init__ on import, so this class does not need to be applied directly.

Parameters:

masker (Optional[SensitiveDataMasker]) – The actual implementation responsible for masking text matching patterns. Note that if a masker is not passed, the MaskingFilter will initialize a masker directly.

filter(record: LogRecord) → bool

Helper method used by the logging.Logger class when adding custom filters to the logging module.

Masks string, dictionary, and list data in log records.

This class will always return True after an attempt to mask sensitive fields is completed.

class scholar_flux.security.MaskingPattern(name: str, pattern: str | SecretStr, *, apply_to_text: bool = True, apply_to_dict: bool = False)

Bases: ABC

The base class for creating MaskingPattern objects that can be used to mask fields based on defined rules.

__init__(*args: Any, **kwargs: Any) → None

abstract apply_masking(text: str) → str

Base method that will be overridden by subclasses to later remove sensitive values and fields from text.

apply_to_dict: bool = False

apply_to_text: bool = True

name: str

pattern: str | SecretStr

class scholar_flux.security.MaskingPatternSet

Bases: set[MaskingPattern]

Subclasses the set to implement the type-safe storage of MaskingPatterns.

As a result, the robustness of operations requiring sensitive pattern masking is increased, reducing the likelihood of unexpected errors during response retrieval and the redaction of sensitive parameters with ScholarFlux.

__init__() → None

Initializes the MaskingPatternSet as an empty set.

add(item: MaskingPattern) → None

Overrides the basic add method to ensure that each item is type-checked prior to entering the set.

update(*others: Iterable[MaskingPattern]) → None

Overrides the basic update method to ensure that all items are type-checked prior to entering the set.

class scholar_flux.security.SecretUtils

Bases: object

Helper utility for both masking and unmasking strings.

Class methods are defined so that they can be used directly or implemented as a mixin so that subclasses can implement the class methods directly.

classmethod is_secret(obj: Any) → bool

Utility class method used to verify whether the current variable is a secret string. This method abstracts the implementation details into a single method to aid further extensibility.

Parameters:

obj (Any) – The object to check

Returns:

True if the object is a SecretStr, False otherwise

Return type:

bool

classmethod mask_secret(obj: None) → None

classmethod mask_secret(obj: Any) → SecretStr

Helper method masking variables into secret strings:

Parameters:

obj (Any | SecretStr) – An object to attempt to unmask if it is a secret string

Returns:

A SecretStr representation of the original object

Return type:

obj (SecretStr)

Examples

>>> from scholar_flux.security import SecretUtils
>>> string = 'a secret'
>>> secret_string = SecretUtils.mask_secret(string)
>>> isinstance(secret_string, SecretStr) is True
# OUTPUT: True

>>> no_string = None
>>> non_secret = SecretUtils.mask_secret(no_string)
>>> non_secret is None
# OUTPUT: True

classmethod unmask_secret(obj: Any) → Any

Helper method for unmasking a variable from a SecretStr into its native type if a secret string.

Parameters:

obj (Any | SecretStr) – An object to attempt to unmask if it is a secret string

Returns:

The object’s original type before being converted into a secret string

Return type:

obj (Any)

Examples

>>> from scholar_flux.security import SecretUtils
>>> from pydantic import SecretStr
>>> string = 'a secret'
>>> secret_string = SecretUtils.mask_secret(string)
>>> isinstance(secret_string, SecretStr) is True
# OUTPUT: True
>>> SecretUtils.unmask_secret(secret_string) == string
# OUTPUT: True
>>> SecretUtils.unmask_secret(None) is None
# OUTPUT: True

class scholar_flux.security.SensitiveDataMasker(register_defaults: bool = True)

Bases: object

The main interface used by the scholar_flux API for masking all text identified as sensitive.

This class is used by scholar_flux to ensure that all sensitive text sent to the scholar_flux.logger is masked.

The SensitiveDataMasker operates through the registration of patterns that identify the text to mask.

Components:

• KeyMaskingPattern:

  identifies specific keys and regex patterns that will signal text to filter

• StringMaskingPattern:

  identifies strings to filter either by fixed or pattern matching

• MaskingPatternSet:

  A customized set accepting only subclasses of MaskingPatterns that specify the rules for filtering text of sensitive fields.

By default, this structure implements masking for email addresses, API keys, bearer tokens, etc. that are identified as sensitive parameters/secrets.

Parameters:

register_defaults (bool) – Determines whether or not to add the patterns that filter API keys email parameters and auth bearers.

Examples

>>> from scholar_flux.security import SensitiveDataMasker # imports the class
>>> masker = SensitiveDataMasker(register_defaults = True) # initializes a masker with defaults
>>> masked = masker.mask_text("'API_KEY' = 'This_Should_Be_Masked_1234', email='a.secret.email@address.com'")
>>> print(masked)
# OUTPUT: 'API_KEY' = '***', email='***'

# specifying a new secret to filter: uses regex by default >>> new_secret = “This string should be filtered” >>> masker.add_sensitive_string_patterns(name=’custom’, patterns=new_secret, use_regex = False)

# applying the filter >>> masked = masker.mask_text(f”The following string should be masked: {new_secret}”) >>> print(masked) # OUTPUT: The following string should be masked: ***

__init__(register_defaults: bool = True) → None

Initializes the SensitiveDataMasker for registering and applying different masking patterns.

Each registered pattern defines a fixed or regular expression pattern to be scrubbed from text with via SensitiveDataMasker.mask_text when matched.

Parameters:

• register_defaults (bool) – Indicates whether to register_defaults for scrubbing emails,

• api_keys

• Bearers (Authorization)

• self.mask_text (etc. from the text when applying)

self.patterns

Indicates the full list of patterns that will be applied when scrubbing text of sensitive fields using masking patterns.

Type:

Set[MaskingPattern]

add_pattern(pattern: MaskingPattern) → None

Adds a pattern to the self.patterns attribute.

add_sensitive_key_patterns(name: str, fields: List[str] | str, fuzzy: bool = False, **kwargs: Any) → None

Adds patterns that identify potentially sensitive strings with the aim of filtering them from logs.

The parameters provided to the method are used to create new string patterns.

Parameters:

• name (str) – The name associated with the pattern (aids identification of patterns)

• fields (List[str] | str) – The list of fields to identify to search and remove from logs.

• pattern (str) – An optional parameter for filtering and removing sensitive fields that match a given pattern. By default this is already set to remove api keys that are typically denoted by alpha numeric fields

• fuzzy (bool) – If true, regular expressions are used to identify keys. Otherwise the fixed (field) key matching is used through the implementation of a basic KeyMaskingPattern.

• **kwargs – Other fields, specifiable via additional keyword arguments that are passed to KeyMaskingPattern

add_sensitive_string_patterns(name: str, patterns: List[str] | str, **kwargs: Any) → None

Adds patterns that identify potentially sensitive strings with the aim of filtering them from logs.

The parameters provided to the method are used to create new string patterns.

Parameters:

• name (str) – The name associated with the pattern (aids identification of patterns)

• patterns (List[str] | str) – The list of patterns to search for and remove from logs

• **kwargs – Other fields, specifiable via additional keyword arguments used to create the StringMaskingPattern

clear() → None

Clears the SensitiveDataMasker.patterns set of all previously registered MaskingPatterns.

This method also clears patterns that were registered by default on the initialization of the SensitiveDataMasker.

The masker would otherwise use the available patterns set to determine what text strings would be masked when the mask_text method is called. Calling mask_text after clearing all MaskingPatterns from the current masker will leave all text unmasked and return the inputted text as is.

get_patterns_by_name(name: str) → Set[MaskingPattern]

Retrieves all patterns with names matching the provided name.

classmethod is_secret(obj: Any) → bool

Utility method for verifying whether the current value is a secret.

This method delegates the verification of the value type to the SecretUtils helper class to abstract the implementation details in cases where the implementation details might require modification in the future for special cases.

Parameters:

obj (Any) – The object to check

Returns:

True if the object is a SecretStr, False otherwise

Return type:

bool

mask_dict(data: dict, convert_objects: bool = False) → dict

Masks sensitive values in dictionaries based on registered key patterns.

This method provides more reliable masking for structured data than string pattern matching, as it directly matches dictionary keys rather than parsing formatted text.

Parameters:

• data (dict) – The dictionary to mask

• convert_objects (bool) – Defines whether to convert data objects (BaseModels, dataclasses) as masked strings

Returns:

New dictionary with sensitive values masked

Return type:

(dict)

Example

>>> masker = SensitiveDataMasker()
>>> config = {'password': 'secret123', 'host': 'localhost'}
>>> masked = masker.mask_dict(config)
>>> print(masked)
# OUTPUT: {'password': '***', 'host': 'localhost'}

mask_list(data: list | tuple, convert_objects: bool = False) → list

Masks sensitive values in lists based on registered patterns.

Recursively processes nested structures (dicts, lists, tuples) and applies masking patterns to any sensitive content found.

Parameters:

data (list | tuple) – list or tuple to mask

Returns:

New list with sensitive values masked

Return type:

list

Note

For type-safety, tuples are converted into lists and may need to be converted as a tuple if used as input.

Example

>>> masker = SensitiveDataMasker()
>>> configs = [{'password': 'secret'}, 'api_key=12345']
>>> masked = masker.mask_list(configs)
>>> print(masked)
# OUTPUT: [{'password': '***'}, 'api_key=***']

mask_object(value: Any) → Any

Converts objects into their masked string representation.

Parameters:

value (Any) – The value to convert and mask

Returns:

Masked string representation of the value

Return type:

str

mask_output(convert_objects: bool = False) → Callable[[Callable[[P], R]], Callable[[P], R]]

Decorator that wraps a function or method using the current SensitiveDataMasker to mask sensitive values.

Parameters:

convert_objects (bool) – If True, objects of unknown types are converted to strings before masking. Otherwise, no conversion takes place, and only strings (passed explicitly or nested within dictionaries, lists, sets) are masked. Defaults to False.

Returns:

A decorator that wraps a function, masking its output according to the registered patterns.

Return type:

Callable[[Callable[P, R]], Callable[P, R]]

Note

The decorated function’s signature and docstring are preserved via functools.wraps.

Example

>>> import os
>>> from scholar_flux import masker
>>> @masker.mask_output()
... def default_config():
...     return {'API_KEY': os.environ.get('EXAMPLE_API_KEY'), 'host': 'https://example-api.com'}
>>> default_config()
# OUTPUT: {'API_KEY': '***', 'host': 'https://example-api.com'}

static mask_secret(obj: None) → None

static mask_secret(obj: Any) → SecretStr

Method for ensuring that any non-secret keys will be masked as secrets.

Parameters:

obj (Any) – An object to attempt to unmask if it is a secret string

Returns:

A SecretStr representation of the original object

Return type:

obj (SecretStr)

mask_text(text: str) → str

Public method for removing sensitive data from text/logs.

Note that the data that is redacted is dependent on what patterns were already previously defined in the SensitiveDataMasker. By default, this includes API keys, emails, and auth headers.

Parameters:

text (str) – the text to scrub of sensitive data

Returns:

The cleaned text that excludes sensitive fields

Return type:

str

mask_value(value: Any, convert_objects: bool = False) → Any

Recursively masks a single value based on its type.

register_secret_if_exists(field: str, value: SecretStr | Any, name: str | None = None, use_regex: bool = False, ignore_case: bool = True) → bool

Identifies fields already registered as secret strings and adds a relevant pattern for ensuring that the field, when unmasked for later use, doesn’t display in logs. Note that if the current field is not a SecretStr, the method will return False without modification or side-effects.

The parameters provided to the method are used to create new string patterns when a SecretStr is detected.

Parameters:

• field (str) – The field, parameter, or key associated with the secret key

• value (SecretStr | Any) – The value, if typed as a secret string, to be registered as a pattern

• name (Optional[str]) – The name to add to identify the relevant pattern by within the pattern set. If not provided, defaults to the field name.

• use_regex (bool) – Indicates whether the current function should use regular expressions when matching the pattern in text. Defaults to False.

• ignore_case (bool) – Whether we should consider case when determining whether or not to filter a string. Defaults to True.

Returns:

If the value is a SecretStr, a string masking pattern is registered for the value and True is returned. if the value is not a SecretStr, False is returned and no side-effects will occur in this case.

Return type:

bool

Example

>>> masker = SensitiveDataMasker()
>>> api_key = SecretStr("sk-123456")
>>> registered = masker.register_secret_if_exists("api_key", api_key)
>>> print(registered)
# OUTPUT: True
>>> registered = masker.register_secret_if_exists("normal_field", "normal_value")
>>> print(registered)
# OUTPUT: False

remove_pattern_by_name(name: str) → int

Removes patterns by name and returns the total number of removed patterns.

structure(flatten: bool = False, show_value_attributes: bool = False) → str

Displays a representation of the current SensitiveDataMasker in a human-readable format.

Parameters:

• flatten (bool) – Indicates whether the SensitiveDataMasker representation should be displayed on a single line.

• show_value_attributes (bool) – Indicates whether the masking patterns should be shown in the console (False by default).

Returns:

A structural representation of the current SensitiveDataMasker

Return type:

str

static unmask_secret(obj: Any) → Any

Method for ensuring that usable values can be successfully extracted from objects.

If the current value is a secret string, this method will return the secret value from the object.

Parameters:

obj (Any) – An object to attempt to unmask if it is a secret string

Returns:

The object’s original type before being converted into a secret string

Return type:

obj (Any)

update(pattern: MaskingPattern | Set[MaskingPattern] | Set[KeyMaskingPattern] | Set[StringMaskingPattern] | MutableSequence[MaskingPattern | KeyMaskingPattern | StringMaskingPattern]) → None

Adds a pattern to the self.patterns attribute.

class scholar_flux.security.StringMaskingPattern(name: str, pattern: str | SecretStr, replacement: str = '***', use_regex: bool = True, ignore_case: bool = True, mask_pattern: bool = True, *, apply_to_text: bool = True, apply_to_dict: bool = False)

Bases: MaskingPattern

Masks values associated with a particular pattern or fixed string in text and API requests.

name

The name to be associated with a particular pattern - can help in later identification and retrieval of rules associated with pattern masks of a particular category.

Type:

str

pattern

The pattern to use to remove sensitive fields, contingent on a parameter being defined. By default, the pattern is set to allow for the removal dashes and alphanumeric fields but can be overridden based on API specific specifications.

Type:

str

replacement

Indicates the replacement string for the value in the string if matched (’***’ by default)

Type:

str

use_regex

Indicates whether the current function should use regular expressions

Type:

bool

ignore_case

whether we should consider case when determining whether or not to filter a string. (True by default)

Type:

bool

mask_pattern

Indicates whether we should, by default, mask pattern strings that are registered in the MaskingPattern. This is True by default.

Type:

bool

apply_to_text

Whether this pattern should be applied during mask_text() operations. Defaults to True.

Type:

bool

apply_to_dict

Whether this pattern should be applied to dictionary values during mask_dict() operations. Defaults to False. String patterns typically match text content (like URLs) rather than dict keys or values.

Type:

bool

Note

By default, this class does not operate on dictionaries unless explicitly configured to do so.

__init__(*args: Any, **kwargs: Any) → None

apply_masking(text: str) → str

Removes sensitive strings or patterns from text based on the defined settings of the StringMaskingPattern.

For the current pattern to trigger, the registered pattern must match the text exactly if use_regex=False. Otherwise, the pattern is treated as a regular expression. Case sensitivity is determined by the ignore_case attribute.

Masking behavior can be further customized based on the defined settings of the current masking pattern based on the attributes specified for pattern, replacement, use_regex, and ignore_case.

Parameters:

text (str) – The text to clean of sensitive fields

Returns:

The text after scrubbing sensitive fields

Return type:

text (str)

ignore_case: bool = True

mask_pattern: bool = True

name: str

pattern: str | SecretStr

replacement: str = '***'

use_regex: bool = True