Some random product page
$ 100.12
''' l = ItemLoader(selector=Selector(html_data)) l.add_xpath('name', '//div[@class="product_name"]/text()') l.add_xpath('name', '//div[@class="product_title"]/text()') l.add_css('price', '#price::text') l.add_value('last_updated', 'today') # you can also use literal values item = l.load_item() item # {'name': ['Some random product page'], 'price': ['$ 100.12'], 'last_updated': ['today']} By quickly looking at that code, we can see the ``name`` field is being extracted from two different XPath locations in the page: 1. ``//div[@class="product_name"]`` 2. ``//div[@class="product_title"]`` In other words, data is being collected by extracting it from two XPath locations, using the :meth:`~ItemLoader.add_xpath` method. This is the data that will be assigned to the ``name`` field later. Afterwards, similar calls are used for ``price`` field using a CSS selector with the :meth:`~ItemLoader.add_css` method, and finally the ``last_update`` field is populated directly with a literal value (``today``) using a different method: :meth:`~ItemLoader.add_value`. Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is called which actually returns the item populated with the data previously extracted and collected with the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css`, and :meth:`~ItemLoader.add_value` calls. Contents -------- .. toctree:: declaring-loaders processors loaders-context nested-loaders extending-loaders built-in-processors api-reference release-notes itemloaders-1.3.1/docs/loaders-context.rst 0000664 0000000 0000000 00000003337 14627331724 0020640 0 ustar 00root root 0000000 0000000 .. currentmodule:: itemloaders .. _loaders-context: Item Loader Context =================== The Item Loader Context is a mechanism that allows to change the input/ouput processors behavior. It's just a ``dict`` of arbitrary key/values which is shared among all processors. By default, the context contains the ``selector`` and any other `keyword arguments` sent to the Loaders's ``__init__``. The context can be passed when declaring, instantiating or using Item Loader. For example, suppose you have a function ``parse_length`` which receives a text value and extracts a length from it:: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a ``loader_context`` argument the function is explicitly telling the Item Loader that it's able to receive an Item Loader context, so the Item Loader passes the currently active context when calling it, and the processor function (``parse_length`` in this case) can thus use them. There are several ways to modify Item Loader context values: 1. By modifying the currently active Item Loader context (:attr:`~ItemLoader.context` attribute):: loader = ItemLoader(product) loader.context['unit'] = 'cm' 2. On Item Loader instantiation (the keyword arguments of Item Loader ``__init__`` method are stored in the Item Loader context):: loader = ItemLoader(product, unit='cm') 3. On Item Loader declaration, for those input/output processors that support instantiating them with an Item Loader context. :class:`~processors.MapCompose` is one of them:: class ProductLoader(ItemLoader): length_out = MapCompose(parse_length, unit='cm') itemloaders-1.3.1/docs/nested-loaders.rst 0000664 0000000 0000000 00000003102 14627331724 0020424 0 ustar 00root root 0000000 0000000 .. _nested-loaders: Nested Loaders ============== When parsing related values from a subsection of a document, it can be useful to create nested loaders. Imagine you're extracting details from a footer of a page that looks something like: Example:: Without nested loaders, you need to specify the full xpath (or css) for each value that you wish to extract. Example:: loader = ItemLoader() # load stuff not in the footer loader.add_xpath('social', '//footer/a[@class = "social"]/@href') loader.add_xpath('email', '//footer/a[@class = "email"]/@href') loader.load_item() Instead, you can create a nested loader with the footer selector and add values relative to the footer. The functionality is the same but you avoid repeating the footer selector. Example:: loader = ItemLoader() # load stuff not in the footer footer_loader = loader.nested_xpath('//footer') footer_loader.add_xpath('social', 'a[@class = "social"]/@href') footer_loader.add_xpath('email', 'a[@class = "email"]/@href') # no need to call footer_loader.load_item() loader.load_item() You can nest loaders arbitrarily and they work with either xpath or css selectors. As a general guideline, use nested loaders when they make your code simpler but do not go overboard with nesting or your parser can become difficult to read. itemloaders-1.3.1/docs/processors.rst 0000664 0000000 0000000 00000006751 14627331724 0017732 0 ustar 00root root 0000000 0000000 .. currentmodule:: itemloaders .. _processors: Input and Output processors =========================== An Item Loader contains one input processor and one output processor for each (item) field. The input processor processes the extracted data as soon as it's received (through the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css` or :meth:`~ItemLoader.add_value` methods) and the result of the input processor is collected and kept inside the ItemLoader. After collecting all data, the :meth:`ItemLoader.load_item` method is called to populate and get the populated item object. That's when the output processor is called with the data previously collected (and processed using the input processor). The result of the output processor is the final value that gets assigned to the item. Let's see an example to illustrate how the input and output processors are called for a particular field (the same applies for any other field):: l = ItemLoader(selector=some_selector) l.add_xpath('name', xpath1) # (1) l.add_xpath('name', xpath2) # (2) l.add_css('name', css) # (3) l.add_value('name', 'test') # (4) return l.load_item() # (5) So what happens is: 1. Data from ``xpath1`` is extracted, and passed through the *input processor* of the ``name`` field. The result of the input processor is collected and kept in the Item Loader (but not yet assigned to the item). 2. Data from ``xpath2`` is extracted, and passed through the same *input processor* used in (1). The result of the input processor is appended to the data collected in (1) (if any). 3. This case is similar to the previous ones, except that the data is extracted from the ``css`` CSS selector, and passed through the same *input processor* used in (1) and (2). The result of the input processor is appended to the data collected in (1) and (2) (if any). 4. This case is also similar to the previous ones, except that the value to be collected is assigned directly, instead of being extracted from a XPath expression or a CSS selector. However, the value is still passed through the input processors. In this case, since the value is not iterable it is converted to an iterable of a single element before passing it to the input processor, because input processor always receive iterables. 5. The data collected in steps (1), (2), (3) and (4) is passed through the *output processor* of the ``name`` field. The result of the output processor is the value assigned to the ``name`` field in the item. It's worth noticing that processors are just callable objects, which are called with the data to be parsed, and return a parsed value. So you can use any function as input or output processor. The only requirement is that they must accept one (and only one) positional argument, which will be an iterable. .. note:: Both input and output processors must receive an iterable as their first argument. The output of those functions can be anything. The result of input processors will be appended to an internal list (in the Loader) containing the collected values (for that field). The result of the output processors is the value that will be finally assigned to the item. The other thing you need to keep in mind is that the values returned by input processors are collected internally (in lists) and then passed to output processors to populate the fields. Last, but not least, ``itemloaders`` comes with some :ref:`commonly used processorsColor TV
loader.add_xpath('name', '//p[@class="product-name"]') # HTML snippet:the price is $1200
loader.add_xpath('price', '//p[@id="price"]', re='the price is (.*)') """ values = self._get_xpathvalues(xpath, **kw) return self.add_value(field_name, values, *processors, re=re, **kw) def replace_xpath( self, field_name: Optional[str], xpath: Union[str, Iterable[str]], *processors: Callable[..., Any], re: Union[str, Pattern[str], None] = None, **kw: Any, ) -> Self: """ Similar to :meth:`add_xpath` but replaces collected data instead of adding it. :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader """ values = self._get_xpathvalues(xpath, **kw) return self.replace_value(field_name, values, *processors, re=re, **kw) def get_xpath( self, xpath: Union[str, Iterable[str]], *processors: Callable[..., Any], re: Union[str, Pattern[str], None] = None, **kw: Any, ) -> Any: """ Similar to :meth:`ItemLoader.get_value` but receives an XPath instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. :param xpath: the XPath to extract data from :type xpath: str :param re: a regular expression to use for extracting data from the selected XPath region :type re: str or typing.Pattern[str] Examples:: # HTML snippet:Color TV
loader.get_xpath('//p[@class="product-name"]') # HTML snippet:the price is $1200
loader.get_xpath('//p[@id="price"]', TakeFirst(), re='the price is (.*)') """ values = self._get_xpathvalues(xpath, **kw) return self.get_value(values, *processors, re=re, **kw) def _get_xpathvalues( self, xpaths: Union[str, Iterable[str]], **kw: Any ) -> List[Any]: self._check_selector_method() assert self.selector is not None xpaths = arg_to_iter(xpaths) return flatten(self.selector.xpath(xpath, **kw).getall() for xpath in xpaths) def add_css( self, field_name: Optional[str], css: Union[str, Iterable[str]], *processors: Callable[..., Any], re: Union[str, Pattern[str], None] = None, **kw: Any, ) -> Self: """ Similar to :meth:`ItemLoader.add_value` but receives a CSS selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. See :meth:`get_css` for ``kwargs``. :param css: the CSS selector to extract data from :type css: str :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader Examples:: # HTML snippet:Color TV
loader.add_css('name', 'p.product-name') # HTML snippet:the price is $1200
loader.add_css('price', 'p#price', re='the price is (.*)') """ values = self._get_cssvalues(css) return self.add_value(field_name, values, *processors, re=re, **kw) def replace_css( self, field_name: Optional[str], css: Union[str, Iterable[str]], *processors: Callable[..., Any], re: Union[str, Pattern[str], None] = None, **kw: Any, ) -> Self: """ Similar to :meth:`add_css` but replaces collected data instead of adding it. :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader """ values = self._get_cssvalues(css) return self.replace_value(field_name, values, *processors, re=re, **kw) def get_css( self, css: Union[str, Iterable[str]], *processors: Callable[..., Any], re: Union[str, Pattern[str], None] = None, **kw: Any, ) -> Any: """ Similar to :meth:`ItemLoader.get_value` but receives a CSS selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. :param css: the CSS selector to extract data from :type css: str :param re: a regular expression to use for extracting data from the selected CSS region :type re: str or typing.Pattern[str] Examples:: # HTML snippet:Color TV
loader.get_css('p.product-name') # HTML snippet:the price is $1200
loader.get_css('p#price', TakeFirst(), re='the price is (.*)') """ values = self._get_cssvalues(css) return self.get_value(values, *processors, re=re, **kw) def _get_cssvalues(self, csss: Union[str, Iterable[str]]) -> List[Any]: self._check_selector_method() assert self.selector is not None csss = arg_to_iter(csss) return flatten(self.selector.css(css).getall() for css in csss) def add_jmes( self, field_name: Optional[str], jmes: str, *processors: Callable[..., Any], re: Union[str, Pattern[str], None] = None, **kw: Any, ) -> Self: """ Similar to :meth:`ItemLoader.add_value` but receives a JMESPath selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. See :meth:`get_jmes` for ``kwargs``. :param jmes: the JMESPath selector to extract data from :type jmes: str :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader Examples:: # HTML snippet: {"name": "Color TV"} loader.add_jmes('name') # HTML snippet: {"price": the price is $1200"} loader.add_jmes('price', TakeFirst(), re='the price is (.*)') """ values = self._get_jmesvalues(jmes) return self.add_value(field_name, values, *processors, re=re, **kw) def replace_jmes( self, field_name: Optional[str], jmes: Union[str, Iterable[str]], *processors: Callable[..., Any], re: Union[str, Pattern[str], None] = None, **kw: Any, ) -> Self: """ Similar to :meth:`add_jmes` but replaces collected data instead of adding it. :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader """ values = self._get_jmesvalues(jmes) return self.replace_value(field_name, values, *processors, re=re, **kw) def get_jmes( self, jmes: Union[str, Iterable[str]], *processors: Callable[..., Any], re: Union[str, Pattern[str], None] = None, **kw: Any, ) -> Any: """ Similar to :meth:`ItemLoader.get_value` but receives a JMESPath selector instead of a value, which is used to extract a list of unicode strings from the selector associated with this :class:`ItemLoader`. :param jmes: the JMESPath selector to extract data from :type jmes: str :param re: a regular expression to use for extracting data from the selected JMESPath :type re: str or typing.Pattern Examples:: # HTML snippet: {"name": "Color TV"} loader.get_jmes('name') # HTML snippet: {"price": the price is $1200"} loader.get_jmes('price', TakeFirst(), re='the price is (.*)') """ values = self._get_jmesvalues(jmes) return self.get_value(values, *processors, re=re, **kw) def _get_jmesvalues(self, jmess: Union[str, Iterable[str]]) -> List[Any]: self._check_selector_method() assert self.selector is not None jmess = arg_to_iter(jmess) if not hasattr(self.selector, "jmespath"): raise AttributeError( "Please install parsel >= 1.8.1 to get jmespath support" ) return flatten(self.selector.jmespath(jmes).getall() for jmes in jmess) itemloaders-1.3.1/itemloaders/common.py 0000664 0000000 0000000 00000001116 14627331724 0020206 0 ustar 00root root 0000000 0000000 """Common functions used in Item Loaders code""" from functools import partial from typing import Any, Callable, MutableMapping from itemloaders.utils import get_func_args def wrap_loader_context( function: Callable[..., Any], context: MutableMapping[str, Any] ) -> Callable[..., Any]: """Wrap functions that receive loader_context to contain the context "pre-loaded" and expose a interface that receives only one argument """ if "loader_context" in get_func_args(function): return partial(function, loader_context=context) else: return function itemloaders-1.3.1/itemloaders/processors.py 0000664 0000000 0000000 00000021532 14627331724 0021124 0 ustar 00root root 0000000 0000000 """ This module provides some commonly used processors for Item Loaders. See documentation in docs/topics/loaders.rst """ from collections import ChainMap from typing import Any, Callable, Iterable, List, MutableMapping, Optional from itemloaders.common import wrap_loader_context from itemloaders.utils import arg_to_iter class MapCompose: """ A processor which is constructed from the composition of the given functions, similar to the :class:`Compose` processor. The difference with this processor is the way internal results are passed among functions, which is as follows: The input value of this processor is *iterated* and the first function is applied to each element. The results of these function calls (one for each element) are concatenated to construct a new iterable, which is then used to apply the second function, and so on, until the last function is applied to each value of the list of values collected so far. The output values of the last function are concatenated together to produce the output of this processor. Each particular function can return a value or a list of values, which is flattened with the list of values returned by the same function applied to the other input values. The functions can also return ``None`` in which case the output of that function is ignored for further processing over the chain. This processor provides a convenient way to compose functions that only work with single values (instead of iterables). For this reason the :class:`MapCompose` processor is typically used as input processor, since data is often extracted using the :meth:`~parsel.selector.Selector.extract` method of `parsel selectors`_, which returns a list of unicode strings. The example below should clarify how it works: >>> def filter_world(x): ... return None if x == 'world' else x ... >>> from itemloaders.processors import MapCompose >>> proc = MapCompose(filter_world, str.upper) >>> proc(['hello', 'world', 'this', 'is', 'something']) ['HELLO', 'THIS', 'IS', 'SOMETHING'] As with the Compose processor, functions can receive Loader contexts, and ``__init__`` method keyword arguments are used as default context values. See :class:`Compose` processor for more info. .. _`parsel selectors`: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.selector.Selector.extract """ # noqa def __init__(self, *functions: Callable[..., Any], **default_loader_context: Any): self.functions = functions self.default_loader_context = default_loader_context def __call__( self, value: Any, loader_context: Optional[MutableMapping[str, Any]] = None ) -> Iterable[Any]: values = arg_to_iter(value) context: MutableMapping[str, Any] if loader_context: context = ChainMap(loader_context, self.default_loader_context) else: context = self.default_loader_context wrapped_funcs = [wrap_loader_context(f, context) for f in self.functions] for func in wrapped_funcs: next_values: List[Any] = [] for v in values: try: next_values += arg_to_iter(func(v)) except Exception as e: raise ValueError( "Error in MapCompose with " "%s value=%r error='%s: %s'" % (str(func), value, type(e).__name__, str(e)) ) from e values = next_values return values class Compose: """ A processor which is constructed from the composition of the given functions. This means that each input value of this processor is passed to the first function, and the result of that function is passed to the second function, and so on, until the last function returns the output value of this processor. By default, stop process on ``None`` value. This behaviour can be changed by passing keyword argument ``stop_on_none=False``. Example: >>> from itemloaders.processors import Compose >>> proc = Compose(lambda v: v[0], str.upper) >>> proc(['hello', 'world']) 'HELLO' Each function can optionally receive a ``loader_context`` parameter. For those which do, this processor will pass the currently active :ref:`Loader context') >>> proc(['one', 'two', 'three']) 'one
two
three' """ def __init__(self, separator: str = " "): self.separator = separator def __call__(self, values: Any) -> str: return self.separator.join(values) itemloaders-1.3.1/itemloaders/py.typed 0000664 0000000 0000000 00000000000 14627331724 0020032 0 ustar 00root root 0000000 0000000 itemloaders-1.3.1/itemloaders/utils.py 0000664 0000000 0000000 00000003015 14627331724 0020056 0 ustar 00root root 0000000 0000000 """ Copy/paste from scrapy source at the moment, to ensure tests are working. Refactoring to come later """ import inspect from functools import partial from typing import Any, Callable, Generator, Iterable, List def arg_to_iter(arg: Any) -> Iterable[Any]: """Return an iterable based on *arg*. If *arg* is a list, a tuple or a generator, it will be returned as is. If *arg* is ``None``, an empty list will be returned. If *arg* is anything else, a list will be returned with *arg* as its only item, i.e. ``[arg]``. """ if arg is None: return [] if isinstance(arg, (list, tuple, Generator)): return arg return [arg] def get_func_args(func: Callable[..., Any], stripself: bool = False) -> List[str]: """Return the argument name list of a callable object""" if not callable(func): raise TypeError(f"func must be callable, got {type(func).__name__!r}") args: List[str] = [] try: sig = inspect.signature(func) except ValueError: return args if isinstance(func, partial): partial_args = func.args partial_kw = func.keywords for name, param in sig.parameters.items(): if param.name in partial_args: continue if partial_kw and param.name in partial_kw: continue args.append(name) else: for name in sig.parameters.keys(): args.append(name) if stripself and args and args[0] == "self": args = args[1:] return args itemloaders-1.3.1/setup.cfg 0000664 0000000 0000000 00000000403 14627331724 0015653 0 ustar 00root root 0000000 0000000 [flake8] ignore = E266, E501, E704, W503 max-line-length = 100 select = B,C,E,F,W,T4,B9 exclude = .git,__pycache__,.venv [isort] profile = black [mypy] [mypy-tests.*] # Allow test functions to be untyped allow_untyped_defs = true check_untyped_defs = true itemloaders-1.3.1/setup.py 0000664 0000000 0000000 00000003336 14627331724 0015554 0 ustar 00root root 0000000 0000000 from setuptools import find_packages, setup with open("README.rst") as f: long_description = f.read() setup( name="itemloaders", version="1.3.1", url="https://github.com/scrapy/itemloaders", project_urls={ "Documentation": "https://itemloaders.readthedocs.io/", "Source": "https://github.com/scrapy/itemloaders", }, description="Base library for scrapy's ItemLoader", long_description=long_description, long_description_content_type="text/x-rst", author="Zyte", author_email="opensource@zyte.com", license="BSD", packages=find_packages(exclude=("tests", "tests.*")), package_data={ "itemadapter": ["py.typed"], }, include_package_data=True, zip_safe=False, classifiers=[ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: BSD License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", ], python_requires=">=3.8", install_requires=[ # before updating these versions, be sure they are not higher than # scrapy's requirements "w3lib>=1.17.0", "parsel>=1.5.0", "jmespath>=0.9.5", "itemadapter>=0.1.0", ], # extras_require=extras_require, ) itemloaders-1.3.1/tests/ 0000775 0000000 0000000 00000000000 14627331724 0015177 5 ustar 00root root 0000000 0000000 itemloaders-1.3.1/tests/__init__.py 0000664 0000000 0000000 00000000000 14627331724 0017276 0 ustar 00root root 0000000 0000000 itemloaders-1.3.1/tests/test_base_loader.py 0000664 0000000 0000000 00000042610 14627331724 0021053 0 ustar 00root root 0000000 0000000 import unittest from functools import partial from itemloaders import ItemLoader from itemloaders.processors import Compose, Identity, Join, MapCompose, TakeFirst class CustomItemLoader(ItemLoader): name_in = MapCompose(lambda v: v.title()) class DefaultedItemLoader(ItemLoader): default_input_processor = MapCompose(lambda v: v[:-1]) # test processors def processor_with_args(value, other=None, loader_context=None): if "key" in loader_context: return loader_context["key"] return value class BasicItemLoaderTest(unittest.TestCase): def test_load_item_using_default_loader(self): i = {"summary": "lala"} il = ItemLoader(item=i) il.add_value("name", "marta") item = il.load_item() assert item is i assert item["summary"] == ["lala"] assert item["name"] == ["marta"] def test_load_item_using_custom_loader(self): il = CustomItemLoader() il.add_value("name", "marta") item = il.load_item() assert item["name"] == ["Marta"] def test_load_item_ignore_none_field_values(self): def validate_sku(value): # Let's assume a SKU is only digits. if value.isdigit(): return value class MyLoader(ItemLoader): name_out = Compose(lambda vs: vs[0]) # take first which allows empty values price_out = Compose(TakeFirst(), float) sku_out = Compose(TakeFirst(), validate_sku) valid_fragment = "SKU: 1234" invalid_fragment = "SKU: not available" sku_re = "SKU: (.+)" il = MyLoader(item={}) # Should not return "sku: None". il.add_value("sku", [invalid_fragment], re=sku_re) # Should not ignore empty values. il.add_value("name", "") il.add_value("price", ["0"]) assert il.load_item() == {"name": "", "price": 0.0} il.replace_value("sku", [valid_fragment], re=sku_re) self.assertEqual(il.load_item()["sku"], "1234") def test_self_referencing_loader(self): class MyLoader(ItemLoader): url_out = TakeFirst() def img_url_out(self, values): return (self.get_output_value("url") or "") + values[0] il = MyLoader(item={}) il.add_value("url", "http://example.com/") il.add_value("img_url", "1234.png") assert il.load_item() == { "url": "http://example.com/", "img_url": "http://example.com/1234.png", } il = MyLoader(item={}) il.add_value("img_url", "1234.png") assert il.load_item() == {"img_url": "1234.png"} def test_add_value(self): il = CustomItemLoader() il.add_value("name", "marta") assert il.get_collected_values("name") == ["Marta"] assert il.get_output_value("name") == ["Marta"] il.add_value("name", "pepe") assert il.get_collected_values("name") == ["Marta", "Pepe"] assert il.get_output_value("name") == ["Marta", "Pepe"] # test add object value il.add_value("summary", {"key": 1}) assert il.get_collected_values("summary") == [{"key": 1}] il.add_value(None, "Jim", lambda x: {"name": x}) assert il.get_collected_values("name") == ["Marta", "Pepe", "Jim"] def test_add_zero(self): il = ItemLoader() il.add_value("name", 0) assert il.get_collected_values("name") == [0] def test_add_none(self): il = ItemLoader() il.add_value("name", None) assert il.get_collected_values("name") == [] def test_replace_value(self): il = CustomItemLoader() il.replace_value("name", "marta") self.assertEqual(il.get_collected_values("name"), ["Marta"]) self.assertEqual(il.get_output_value("name"), ["Marta"]) il.replace_value("name", "pepe") self.assertEqual(il.get_collected_values("name"), ["Pepe"]) self.assertEqual(il.get_output_value("name"), ["Pepe"]) il.replace_value(None, "Jim", lambda x: {"name": x}) self.assertEqual(il.get_collected_values("name"), ["Jim"]) def test_replace_value_none(self): il = CustomItemLoader() il.replace_value("name", None) self.assertEqual(il.get_collected_values("name"), []) il.replace_value("name", "marta") self.assertEqual(il.get_collected_values("name"), ["Marta"]) il.replace_value( "name", None ) # when replacing with `None` nothing should happen self.assertEqual(il.get_collected_values("name"), ["Marta"]) def test_get_value(self): il = ItemLoader() self.assertEqual("FOO", il.get_value(["foo", "bar"], TakeFirst(), str.upper)) self.assertEqual( ["foo", "bar"], il.get_value(["name:foo", "name:bar"], re="name:(.*)$") ) self.assertEqual( "foo", il.get_value(["name:foo", "name:bar"], TakeFirst(), re="name:(.*)$") ) self.assertEqual( None, il.get_value(["foo", "bar"], TakeFirst(), re="name:(.*)$") ) self.assertEqual(None, il.get_value(None, TakeFirst())) il.add_value("name", ["name:foo", "name:bar"], TakeFirst(), re="name:(.*)$") self.assertEqual(["foo"], il.get_collected_values("name")) il.replace_value("name", "name:bar", re="name:(.*)$") self.assertEqual(["bar"], il.get_collected_values("name")) def test_iter_on_input_processor_input(self): class NameFirstItemLoader(ItemLoader): name_in = TakeFirst() il = NameFirstItemLoader() il.add_value("name", "marta") self.assertEqual(il.get_collected_values("name"), ["marta"]) il = NameFirstItemLoader() il.add_value("name", ["marta", "jose"]) self.assertEqual(il.get_collected_values("name"), ["marta"]) il = NameFirstItemLoader() il.replace_value("name", "marta") self.assertEqual(il.get_collected_values("name"), ["marta"]) il = NameFirstItemLoader() il.replace_value("name", ["marta", "jose"]) self.assertEqual(il.get_collected_values("name"), ["marta"]) il = NameFirstItemLoader() il.add_value("name", "marta") il.add_value("name", ["jose", "pedro"]) self.assertEqual(il.get_collected_values("name"), ["marta", "jose"]) def test_map_compose_filter(self): def filter_world(x): return None if x == "world" else x proc = MapCompose(filter_world, str.upper) self.assertEqual( proc(["hello", "world", "this", "is", "scrapy"]), ["HELLO", "THIS", "IS", "SCRAPY"], ) def test_map_compose_filter_multil(self): class CustomItemLoader(ItemLoader): name_in = MapCompose(lambda v: v.title(), lambda v: v[:-1]) il = CustomItemLoader() il.add_value("name", "marta") self.assertEqual(il.get_output_value("name"), ["Mart"]) item = il.load_item() self.assertEqual(item["name"], ["Mart"]) def test_default_input_processor(self): il = DefaultedItemLoader() il.add_value("name", "marta") self.assertEqual(il.get_output_value("name"), ["mart"]) def test_inherited_default_input_processor(self): class InheritDefaultedItemLoader(DefaultedItemLoader): pass il = InheritDefaultedItemLoader() il.add_value("name", "marta") self.assertEqual(il.get_output_value("name"), ["mart"]) def test_input_processor_inheritance(self): class ChildItemLoader(CustomItemLoader): url_in = MapCompose(lambda v: v.lower()) il = ChildItemLoader() il.add_value("url", "HTTP://scrapy.ORG") self.assertEqual(il.get_output_value("url"), ["http://scrapy.org"]) il.add_value("name", "marta") self.assertEqual(il.get_output_value("name"), ["Marta"]) class ChildChildItemLoader(ChildItemLoader): url_in = MapCompose(lambda v: v.upper()) summary_in = MapCompose(lambda v: v) il = ChildChildItemLoader() il.add_value("url", "http://scrapy.org") self.assertEqual(il.get_output_value("url"), ["HTTP://SCRAPY.ORG"]) il.add_value("name", "marta") self.assertEqual(il.get_output_value("name"), ["Marta"]) def test_empty_map_compose(self): class IdentityDefaultedItemLoader(DefaultedItemLoader): name_in = MapCompose() il = IdentityDefaultedItemLoader() il.add_value("name", "marta") self.assertEqual(il.get_output_value("name"), ["marta"]) def test_identity_input_processor(self): class IdentityDefaultedItemLoader(DefaultedItemLoader): name_in = Identity() il = IdentityDefaultedItemLoader() il.add_value("name", "marta") self.assertEqual(il.get_output_value("name"), ["marta"]) def test_extend_custom_input_processors(self): class ChildItemLoader(CustomItemLoader): name_in = MapCompose(CustomItemLoader.name_in, str.swapcase) il = ChildItemLoader() il.add_value("name", "marta") self.assertEqual(il.get_output_value("name"), ["mARTA"]) def test_extend_default_input_processors(self): class ChildDefaultedItemLoader(DefaultedItemLoader): name_in = MapCompose( DefaultedItemLoader.default_input_processor, str.swapcase ) il = ChildDefaultedItemLoader() il.add_value("name", "marta") self.assertEqual(il.get_output_value("name"), ["MART"]) def test_output_processor_using_function(self): il = CustomItemLoader() il.add_value("name", ["mar", "ta"]) self.assertEqual(il.get_output_value("name"), ["Mar", "Ta"]) class TakeFirstItemLoader(CustomItemLoader): name_out = " ".join il = TakeFirstItemLoader() il.add_value("name", ["mar", "ta"]) self.assertEqual(il.get_output_value("name"), "Mar Ta") def test_output_processor_error(self): class CustomItemLoader(ItemLoader): name_out = MapCompose(float) il = CustomItemLoader() il.add_value("name", ["$10"]) try: float("$10") except Exception as e: expected_exc_str = str(e) exc = None try: il.load_item() except Exception as e: exc = e assert isinstance(exc, ValueError) s = str(exc) assert "name" in s, s assert "$10" in s, s assert "ValueError" in s, s assert expected_exc_str in s, s def test_output_processor_using_classes(self): il = CustomItemLoader() il.add_value("name", ["mar", "ta"]) self.assertEqual(il.get_output_value("name"), ["Mar", "Ta"]) class TakeFirstItemLoader1(CustomItemLoader): name_out = Join() il = TakeFirstItemLoader1() il.add_value("name", ["mar", "ta"]) self.assertEqual(il.get_output_value("name"), "Mar Ta") class TakeFirstItemLoader2(CustomItemLoader): name_out = Join("
") il = TakeFirstItemLoader2() il.add_value("name", ["mar", "ta"]) self.assertEqual(il.get_output_value("name"), "Mar
Ta") def test_default_output_processor(self): il = CustomItemLoader() il.add_value("name", ["mar", "ta"]) self.assertEqual(il.get_output_value("name"), ["Mar", "Ta"]) class LalaItemLoader(CustomItemLoader): default_output_processor = Identity() il = LalaItemLoader() il.add_value("name", ["mar", "ta"]) self.assertEqual(il.get_output_value("name"), ["Mar", "Ta"]) def test_loader_context_on_declaration(self): class ChildItemLoader(CustomItemLoader): url_in = MapCompose(processor_with_args, key="val") il = ChildItemLoader() il.add_value("url", "text") self.assertEqual(il.get_output_value("url"), ["val"]) il.replace_value("url", "text2") self.assertEqual(il.get_output_value("url"), ["val"]) def test_loader_context_on_instantiation(self): class ChildItemLoader(CustomItemLoader): url_in = MapCompose(processor_with_args) il = ChildItemLoader(key="val") il.add_value("url", "text") self.assertEqual(il.get_output_value("url"), ["val"]) il.replace_value("url", "text2") self.assertEqual(il.get_output_value("url"), ["val"]) def test_loader_context_on_assign(self): class ChildItemLoader(CustomItemLoader): url_in = MapCompose(processor_with_args) il = ChildItemLoader() il.context["key"] = "val" il.add_value("url", "text") self.assertEqual(il.get_output_value("url"), ["val"]) il.replace_value("url", "text2") self.assertEqual(il.get_output_value("url"), ["val"]) def test_item_passed_to_input_processor_functions(self): def processor(value, loader_context): return loader_context["item"]["name"] class ChildItemLoader(CustomItemLoader): url_in = MapCompose(processor) it = {"name": "marta"} il = ChildItemLoader(item=it) il.add_value("url", "text") self.assertEqual(il.get_output_value("url"), ["marta"]) il.replace_value("url", "text2") self.assertEqual(il.get_output_value("url"), ["marta"]) # def test_add_value_on_unknown_field(self): # il = CustomItemLoader() # self.assertRaises(KeyError, il.add_value, 'wrong_field', ['lala', 'lolo']) def test_compose_processor(self): class CustomItemLoader(ItemLoader): name_out = Compose(lambda v: v[0], lambda v: v.title(), lambda v: v[:-1]) il = CustomItemLoader() il.add_value("name", ["marta", "other"]) self.assertEqual(il.get_output_value("name"), "Mart") item = il.load_item() self.assertEqual(item["name"], "Mart") def test_partial_processor(self): def join(values, sep=None, loader_context=None, ignored=None): if sep is not None: return sep.join(values) elif loader_context and "sep" in loader_context: return loader_context["sep"].join(values) else: return "".join(values) class CustomItemLoader(ItemLoader): name_out = Compose(partial(join, sep="+")) url_out = Compose(partial(join, loader_context={"sep": "."})) summary_out = Compose(partial(join, ignored="foo")) il = CustomItemLoader() il.add_value("name", ["rabbit", "hole"]) il.add_value("url", ["rabbit", "hole"]) il.add_value("summary", ["rabbit", "hole"]) item = il.load_item() self.assertEqual(item["name"], "rabbit+hole") self.assertEqual(item["url"], "rabbit.hole") self.assertEqual(item["summary"], "rabbithole") def test_error_input_processor(self): class CustomItemLoader(ItemLoader): name_in = MapCompose(float) il = CustomItemLoader() self.assertRaises(ValueError, il.add_value, "name", ["marta", "other"]) def test_error_output_processor(self): class CustomItemLoader(ItemLoader): name_out = Compose(Join(), float) il = CustomItemLoader() il.add_value("name", "marta") with self.assertRaises(ValueError): il.load_item() def test_error_processor_as_argument(self): il = CustomItemLoader() self.assertRaises( ValueError, il.add_value, "name", ["marta", "other"], Compose(float) ) def test_get_unset_value(self): loader = ItemLoader() self.assertEqual(loader.load_item(), {}) self.assertEqual(loader.get_output_value("foo"), []) self.assertEqual(loader.load_item(), {}) class BaseNoInputReprocessingLoader(ItemLoader): title_in = MapCompose(str.upper) title_out = TakeFirst() class NoInputReprocessingDictLoader(BaseNoInputReprocessingLoader): default_item_class = dict class NoInputReprocessingFromDictTest(unittest.TestCase): """ Loaders initialized from loaded items must not reprocess fields (dict instances) """ def test_avoid_reprocessing_with_initial_values_single(self): il = NoInputReprocessingDictLoader(item={"title": "foo"}) il_loaded = il.load_item() self.assertEqual(il_loaded, {"title": "foo"}) self.assertEqual( NoInputReprocessingDictLoader(item=il_loaded).load_item(), {"title": "foo"} ) def test_avoid_reprocessing_with_initial_values_list(self): il = NoInputReprocessingDictLoader(item={"title": ["foo", "bar"]}) il_loaded = il.load_item() self.assertEqual(il_loaded, {"title": "foo"}) self.assertEqual( NoInputReprocessingDictLoader(item=il_loaded).load_item(), {"title": "foo"} ) def test_avoid_reprocessing_without_initial_values_single(self): il = NoInputReprocessingDictLoader() il.add_value("title", "foo") il_loaded = il.load_item() self.assertEqual(il_loaded, {"title": "FOO"}) self.assertEqual( NoInputReprocessingDictLoader(item=il_loaded).load_item(), {"title": "FOO"} ) def test_avoid_reprocessing_without_initial_values_list(self): il = NoInputReprocessingDictLoader() il.add_value("title", ["foo", "bar"]) il_loaded = il.load_item() self.assertEqual(il_loaded, {"title": "FOO"}) self.assertEqual( NoInputReprocessingDictLoader(item=il_loaded).load_item(), {"title": "FOO"} ) itemloaders-1.3.1/tests/test_loader_initialization.py 0000664 0000000 0000000 00000011115 14627331724 0023164 0 ustar 00root root 0000000 0000000 import unittest from typing import Any, Protocol from itemloaders import ItemLoader class InitializationTestProtocol(Protocol): item_class: Any def assertEqual(self, first: Any, second: Any, msg: Any = ...) -> None: ... def assertIsInstance(self, obj: object, cls: type, msg: Any = None) -> None: ... class InitializationTestMixin: item_class: Any = None def test_keep_single_value(self: InitializationTestProtocol) -> None: """Loaded item should contain values from the initial item""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) loaded_item = il.load_item() self.assertIsInstance(loaded_item, self.item_class) self.assertEqual(dict(loaded_item), {"name": ["foo"]}) def test_keep_list(self: InitializationTestProtocol) -> None: """Loaded item should contain values from the initial item""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) loaded_item = il.load_item() self.assertIsInstance(loaded_item, self.item_class) self.assertEqual(dict(loaded_item), {"name": ["foo", "bar"]}) def test_add_value_singlevalue_singlevalue( self: InitializationTestProtocol, ) -> None: """Values added after initialization should be appended""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) il.add_value("name", "bar") loaded_item = il.load_item() self.assertIsInstance(loaded_item, self.item_class) self.assertEqual(dict(loaded_item), {"name": ["foo", "bar"]}) def test_add_value_singlevalue_list(self: InitializationTestProtocol) -> None: """Values added after initialization should be appended""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) il.add_value("name", ["item", "loader"]) loaded_item = il.load_item() self.assertIsInstance(loaded_item, self.item_class) self.assertEqual(dict(loaded_item), {"name": ["foo", "item", "loader"]}) def test_add_value_list_singlevalue(self: InitializationTestProtocol) -> None: """Values added after initialization should be appended""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) il.add_value("name", "qwerty") loaded_item = il.load_item() self.assertIsInstance(loaded_item, self.item_class) self.assertEqual(dict(loaded_item), {"name": ["foo", "bar", "qwerty"]}) def test_add_value_list_list(self: InitializationTestProtocol) -> None: """Values added after initialization should be appended""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) il.add_value("name", ["item", "loader"]) loaded_item = il.load_item() self.assertIsInstance(loaded_item, self.item_class) self.assertEqual(dict(loaded_item), {"name": ["foo", "bar", "item", "loader"]}) def test_get_output_value_singlevalue(self: InitializationTestProtocol) -> None: """Getting output value must not remove value from item""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) self.assertEqual(il.get_output_value("name"), ["foo"]) loaded_item = il.load_item() self.assertIsInstance(loaded_item, self.item_class) self.assertEqual(loaded_item, {"name": ["foo"]}) def test_get_output_value_list(self: InitializationTestProtocol) -> None: """Getting output value must not remove value from item""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) self.assertEqual(il.get_output_value("name"), ["foo", "bar"]) loaded_item = il.load_item() self.assertIsInstance(loaded_item, self.item_class) self.assertEqual(loaded_item, {"name": ["foo", "bar"]}) def test_values_single(self: InitializationTestProtocol) -> None: """Values from initial item must be added to loader._values""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) self.assertEqual(il._values.get("name"), ["foo"]) def test_values_list(self: InitializationTestProtocol) -> None: """Values from initial item must be added to loader._values""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) self.assertEqual(il._values.get("name"), ["foo", "bar"]) class InitializationFromDictTest(InitializationTestMixin, unittest.TestCase): item_class = dict itemloaders-1.3.1/tests/test_nested_items.py 0000664 0000000 0000000 00000002445 14627331724 0021300 0 ustar 00root root 0000000 0000000 import unittest from typing import Any from itemloaders import ItemLoader class NestedItemTest(unittest.TestCase): """Test that adding items as values works as expected.""" def _test_item(self, item: Any) -> None: il = ItemLoader() il.add_value("item_list", item) self.assertEqual(il.load_item(), {"item_list": [item]}) def test_attrs(self): try: import attr except ImportError: self.skipTest("Cannot import attr") @attr.s class TestItem: foo = attr.ib() self._test_item(TestItem(foo="bar")) def test_dataclass(self): try: from dataclasses import dataclass except ImportError: self.skipTest("Cannot import dataclasses.dataclass") @dataclass class TestItem: foo: str self._test_item(TestItem(foo="bar")) def test_dict(self): self._test_item({"foo": "bar"}) def test_scrapy_item(self): try: from scrapy import Field, Item except ImportError: self.skipTest("Cannot import Field or Item from scrapy") # needs py.typed in Scrapy class TestItem(Item): # type: ignore[misc] foo = Field() self._test_item(TestItem(foo="bar")) itemloaders-1.3.1/tests/test_nested_loader.py 0000664 0000000 0000000 00000010573 14627331724 0021426 0 ustar 00root root 0000000 0000000 import unittest from parsel import Selector from itemloaders import ItemLoader class SubselectorLoaderTest(unittest.TestCase): selector = Selector( text="""
marta
paragraph
marta
']
)
self.assertEqual(loader.get_output_value("name_value"), ["marta"])
self.assertEqual(loader.get_output_value("name"), nl.get_output_value("name"))
self.assertEqual(
loader.get_output_value("name_div"), nl.get_output_value("name_div")
)
self.assertEqual(
loader.get_output_value("name_value"), nl.get_output_value("name_value")
)
def test_nested_css(self):
loader = ItemLoader(selector=self.selector)
nl = loader.nested_css("header")
nl.add_xpath("name", "div/text()")
nl.add_css("name_div", "#id")
assert nl.selector
nl.add_value("name_value", nl.selector.xpath('div[@id = "id"]/text()').getall())
self.assertEqual(loader.get_output_value("name"), ["marta"])
self.assertEqual(
loader.get_output_value("name_div"), ['marta
']
)
self.assertEqual(loader.get_output_value("name_value"), ["marta"])
self.assertEqual(loader.get_output_value("name"), nl.get_output_value("name"))
self.assertEqual(
loader.get_output_value("name_div"), nl.get_output_value("name_div")
)
self.assertEqual(
loader.get_output_value("name_value"), nl.get_output_value("name_value")
)
def test_nested_replace(self):
loader = ItemLoader(selector=self.selector)
nl1 = loader.nested_xpath("//footer")
nl2 = nl1.nested_xpath("a")
loader.add_xpath("url", "//footer/a/@href")
self.assertEqual(loader.get_output_value("url"), ["http://www.scrapy.org"])
nl1.replace_xpath("url", "img/@src")
self.assertEqual(loader.get_output_value("url"), ["/images/logo.png"])
nl2.replace_xpath("url", "@href")
self.assertEqual(loader.get_output_value("url"), ["http://www.scrapy.org"])
def test_nested_ordering(self):
loader = ItemLoader(selector=self.selector)
nl1 = loader.nested_xpath("//footer")
nl2 = nl1.nested_xpath("a")
nl1.add_xpath("url", "img/@src")
loader.add_xpath("url", "//footer/a/@href")
nl2.add_xpath("url", "text()")
loader.add_xpath("url", "//footer/a/@href")
self.assertEqual(
loader.get_output_value("url"),
[
"/images/logo.png",
"http://www.scrapy.org",
"homepage",
"http://www.scrapy.org",
],
)
def test_nested_load_item(self):
loader = ItemLoader(selector=self.selector)
nl1 = loader.nested_xpath("//footer")
nl2 = nl1.nested_xpath("img")
loader.add_xpath("name", "//header/div/text()")
nl1.add_xpath("url", "a/@href")
nl2.add_xpath("image", "@src")
item = loader.load_item()
assert item is loader.item
assert item is nl1.item
assert item is nl2.item
self.assertEqual(item["name"], ["marta"])
self.assertEqual(item["url"], ["http://www.scrapy.org"])
self.assertEqual(item["image"], ["/images/logo.png"])
def test_nested_empty_selector(self):
loader = ItemLoader(selector=self.selector)
nested_xpath = loader.nested_xpath("//bar")
assert isinstance(nested_xpath, ItemLoader)
nested_xpath.add_xpath("foo", "./foo")
nested_css = loader.nested_css("bar")
assert isinstance(nested_css, ItemLoader)
nested_css.add_css("foo", "foo")
itemloaders-1.3.1/tests/test_output_processor.py 0000664 0000000 0000000 00000002362 14627331724 0022252 0 ustar 00root root 0000000 0000000 import unittest
from typing import Any, Dict
from itemloaders import ItemLoader
from itemloaders.processors import Compose, Identity, TakeFirst
class TestOutputProcessorDict(unittest.TestCase):
def test_output_processor(self):
class TempDict(Dict[str, Any]):
def __init__(self, *args, **kwargs):
super(TempDict, self).__init__(self, *args, **kwargs)
self.setdefault("temp", 0.3)
class TempLoader(ItemLoader):
default_item_class = TempDict
default_input_processor = Identity()
default_output_processor = Compose(TakeFirst())
loader = TempLoader()
item = loader.load_item()
self.assertIsInstance(item, TempDict)
self.assertEqual(dict(item), {"temp": 0.3})
class TestOutputProcessorItem(unittest.TestCase):
def test_output_processor(self):
class TempLoader(ItemLoader):
default_input_processor = Identity()
default_output_processor = Compose(TakeFirst())
item: Dict[str, Any] = {}
item.setdefault("temp", 0.3)
loader = TempLoader(item=item)
item = loader.load_item()
self.assertIsInstance(item, dict)
self.assertEqual(dict(item), {"temp": 0.3})
itemloaders-1.3.1/tests/test_processors.py 0000664 0000000 0000000 00000003500 14627331724 0021010 0 ustar 00root root 0000000 0000000 import unittest
from itemloaders.processors import Compose, Identity, Join, MapCompose, TakeFirst
class ProcessorsTest(unittest.TestCase):
def test_take_first(self):
proc = TakeFirst()
self.assertEqual(proc([None, "", "hello", "world"]), "hello")
self.assertEqual(proc([None, "", 0, "hello", "world"]), 0)
def test_identity(self):
proc = Identity()
self.assertEqual(
proc([None, "", "hello", "world"]), [None, "", "hello", "world"]
)
def test_join(self):
proc = Join()
self.assertRaises(TypeError, proc, [None, "", "hello", "world"])
self.assertEqual(proc(["", "hello", "world"]), " hello world")
self.assertEqual(proc(["hello", "world"]), "hello world")
self.assertIsInstance(proc(["hello", "world"]), str)
def test_compose(self):
proc = Compose(lambda v: v[0], str.upper)
self.assertEqual(proc(["hello", "world"]), "HELLO")
proc = Compose(str.upper)
self.assertEqual(proc(None), None)
proc = Compose(str.upper, stop_on_none=False)
self.assertRaises(ValueError, proc, None)
proc = Compose(str.upper, lambda x: x + 1)
self.assertRaises(ValueError, proc, "hello")
def test_mapcompose(self):
def filter_world(x):
return None if x == "world" else x
proc = MapCompose(filter_world, str.upper)
self.assertEqual(
proc(["hello", "world", "this", "is", "scrapy"]),
["HELLO", "THIS", "IS", "SCRAPY"],
)
proc = MapCompose(filter_world, str.upper)
self.assertEqual(proc(None), [])
proc = MapCompose(filter_world, str.upper)
self.assertRaises(ValueError, proc, [1])
proc = MapCompose(filter_world, lambda x: x + 1)
self.assertRaises(ValueError, proc, "hello")
itemloaders-1.3.1/tests/test_select_jmes.py 0000664 0000000 0000000 00000001641 14627331724 0021107 0 ustar 00root root 0000000 0000000 import unittest
from itemloaders.processors import SelectJmes
class SelectJmesTestCase(unittest.TestCase):
test_list_equals = {
"simple": ("foo.bar", {"foo": {"bar": "baz"}}, "baz"),
"invalid": ("foo.bar.baz", {"foo": {"bar": "baz"}}, None),
"top_level": ("foo", {"foo": {"bar": "baz"}}, {"bar": "baz"}),
"double_vs_single_quote_string": ("foo.bar", {"foo": {"bar": "baz"}}, "baz"),
"dict": (
"foo.bar[*].name",
{"foo": {"bar": [{"name": "one"}, {"name": "two"}]}},
["one", "two"],
),
"list": ("[1]", [1, 2], 2),
}
def test_output(self):
for key in self.test_list_equals:
expr, test_list, expected = self.test_list_equals[key]
test = SelectJmes(expr)(test_list)
self.assertEqual(
test, expected, msg=f"test {key!r} got {test} expected {expected}"
)
itemloaders-1.3.1/tests/test_selector_loader.py 0000664 0000000 0000000 00000030434 14627331724 0021762 0 ustar 00root root 0000000 0000000 import re
import unittest
from unittest.mock import MagicMock
from parsel import Selector
from itemloaders import ItemLoader
from itemloaders.processors import MapCompose, TakeFirst
class CustomItemLoader(ItemLoader):
name_in = MapCompose(lambda v: v.title())
class SelectortemLoaderTest(unittest.TestCase):
selector = Selector(
text="""
marta
paragraph
homepage
"""
)
jmes_selector = Selector(
text="""
{
"name": "marta",
"description": "paragraph",
"website": {
"url": "http://www.scrapy.org",
"name": "homepage"
},
"logo": "/images/logo.png"
}
"""
)
def test_init_method(self):
loader = CustomItemLoader()
self.assertEqual(loader.selector, None)
def test_init_method_errors(self):
loader = CustomItemLoader()
self.assertRaises(RuntimeError, loader.add_xpath, "url", "//a/@href")
self.assertRaises(RuntimeError, loader.replace_xpath, "url", "//a/@href")
self.assertRaises(RuntimeError, loader.get_xpath, "//a/@href")
self.assertRaises(RuntimeError, loader.add_css, "name", "#name::text")
self.assertRaises(RuntimeError, loader.replace_css, "name", "#name::text")
self.assertRaises(RuntimeError, loader.get_css, "#name::text")
def test_init_method_with_selector(self):
loader = CustomItemLoader(selector=self.selector)
self.assertTrue(loader.selector)
loader.add_xpath("name", "//div/text()")
self.assertEqual(loader.get_output_value("name"), ["Marta"])
def test_init_method_with_selector_css(self):
loader = CustomItemLoader(selector=self.selector)
self.assertTrue(loader.selector)
loader.add_css("name", "div::text")
self.assertEqual(loader.get_output_value("name"), ["Marta"])
loader.add_css("url", "a::attr(href)")
self.assertEqual(loader.get_output_value("url"), ["http://www.scrapy.org"])
# combining/accumulating CSS selectors and XPath expressions
loader.add_xpath("name", "//div/text()")
self.assertEqual(loader.get_output_value("name"), ["Marta", "Marta"])
loader.add_xpath("url", "//img/@src")
self.assertEqual(
loader.get_output_value("url"),
["http://www.scrapy.org", "/images/logo.png"],
)
def test_add_xpath_re(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath("name", "//div/text()", re="ma")
self.assertEqual(loader.get_output_value("name"), ["Ma"])
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath("name", "//div/text()", re=re.compile("ma"))
self.assertEqual(loader.get_output_value("name"), ["Ma"])
def test_add_xpath_variables(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath("name", "id($id)/text()", id="id")
self.assertEqual(loader.get_output_value("name"), ["Marta"])
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath("name", "id($id)/text()", id="id2")
self.assertEqual(loader.get_output_value("name"), [])
def test_replace_xpath(self):
loader = CustomItemLoader(selector=self.selector)
self.assertTrue(loader.selector)
loader.add_xpath("name", "//div/text()")
self.assertEqual(loader.get_output_value("name"), ["Marta"])
loader.replace_xpath("name", "//p/text()")
self.assertEqual(loader.get_output_value("name"), ["Paragraph"])
loader.replace_xpath("name", ["//p/text()", "//div/text()"])
self.assertEqual(loader.get_output_value("name"), ["Paragraph", "Marta"])
def test_get_xpath(self):
loader = CustomItemLoader(selector=self.selector)
self.assertEqual(loader.get_xpath("//p/text()"), ["paragraph"])
self.assertEqual(loader.get_xpath("//p/text()", TakeFirst()), "paragraph")
self.assertEqual(loader.get_xpath("//p/text()", TakeFirst(), re="pa"), "pa")
self.assertEqual(
loader.get_xpath(["//p/text()", "//div/text()"]), ["paragraph", "marta"]
)
def test_replace_xpath_multi_fields(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath(None, "//div/text()", TakeFirst(), lambda x: {"name": x})
self.assertEqual(loader.get_output_value("name"), ["Marta"])
loader.replace_xpath(None, "//p/text()", TakeFirst(), lambda x: {"name": x})
self.assertEqual(loader.get_output_value("name"), ["Paragraph"])
def test_replace_xpath_re(self):
loader = CustomItemLoader(selector=self.selector)
self.assertTrue(loader.selector)
loader.add_xpath("name", "//div/text()")
self.assertEqual(loader.get_output_value("name"), ["Marta"])
loader.replace_xpath("name", "//div/text()", re="ma")
self.assertEqual(loader.get_output_value("name"), ["Ma"])
def test_add_css_re(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_css("name", "div::text", re="ma")
self.assertEqual(loader.get_output_value("name"), ["Ma"])
loader.add_css("url", "a::attr(href)", re="http://(.+)")
self.assertEqual(loader.get_output_value("url"), ["www.scrapy.org"])
loader = CustomItemLoader(selector=self.selector)
loader.add_css("name", "div::text", re=re.compile("ma"))
self.assertEqual(loader.get_output_value("name"), ["Ma"])
loader.add_css("url", "a::attr(href)", re=re.compile("http://(.+)"))
self.assertEqual(loader.get_output_value("url"), ["www.scrapy.org"])
def test_replace_css(self):
loader = CustomItemLoader(selector=self.selector)
self.assertTrue(loader.selector)
loader.add_css("name", "div::text")
self.assertEqual(loader.get_output_value("name"), ["Marta"])
loader.replace_css("name", "p::text")
self.assertEqual(loader.get_output_value("name"), ["Paragraph"])
loader.replace_css("name", ["p::text", "div::text"])
self.assertEqual(loader.get_output_value("name"), ["Paragraph", "Marta"])
loader.add_css("url", "a::attr(href)", re="http://(.+)")
self.assertEqual(loader.get_output_value("url"), ["www.scrapy.org"])
loader.replace_css("url", "img::attr(src)")
self.assertEqual(loader.get_output_value("url"), ["/images/logo.png"])
def test_get_css(self):
loader = CustomItemLoader(selector=self.selector)
self.assertEqual(loader.get_css("p::text"), ["paragraph"])
self.assertEqual(loader.get_css("p::text", TakeFirst()), "paragraph")
self.assertEqual(loader.get_css("p::text", TakeFirst(), re="pa"), "pa")
self.assertEqual(
loader.get_css(["p::text", "div::text"]), ["paragraph", "marta"]
)
self.assertEqual(
loader.get_css(["a::attr(href)", "img::attr(src)"]),
["http://www.scrapy.org", "/images/logo.png"],
)
def test_replace_css_multi_fields(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_css(None, "div::text", TakeFirst(), lambda x: {"name": x})
self.assertEqual(loader.get_output_value("name"), ["Marta"])
loader.replace_css(None, "p::text", TakeFirst(), lambda x: {"name": x})
self.assertEqual(loader.get_output_value("name"), ["Paragraph"])
loader.add_css(None, "a::attr(href)", TakeFirst(), lambda x: {"url": x})
self.assertEqual(loader.get_output_value("url"), ["http://www.scrapy.org"])
loader.replace_css(None, "img::attr(src)", TakeFirst(), lambda x: {"url": x})
self.assertEqual(loader.get_output_value("url"), ["/images/logo.png"])
def test_replace_css_re(self):
loader = CustomItemLoader(selector=self.selector)
self.assertTrue(loader.selector)
loader.add_css("url", "a::attr(href)")
self.assertEqual(loader.get_output_value("url"), ["http://www.scrapy.org"])
loader.replace_css("url", "a::attr(href)", re=r"http://www\.(.+)")
self.assertEqual(loader.get_output_value("url"), ["scrapy.org"])
def test_jmes_not_installed(self):
selector = MagicMock(spec=Selector)
del selector.jmespath
loader = CustomItemLoader(selector=selector)
with self.assertRaises(AttributeError) as err:
loader.add_jmes("name", "name", re="ma")
self.assertEqual(
str(err.exception), "Please install parsel >= 1.8.1 to get jmespath support"
)
def test_add_jmes_re(self):
loader = CustomItemLoader(selector=self.jmes_selector)
loader.add_jmes("name", "name", re="ma")
self.assertEqual(loader.get_output_value("name"), ["Ma"])
loader.add_jmes("url", "website.url", re="http://(.+)")
self.assertEqual(loader.get_output_value("url"), ["www.scrapy.org"])
loader = CustomItemLoader(selector=self.jmes_selector)
loader.add_jmes("name", "name", re=re.compile("ma"))
self.assertEqual(loader.get_output_value("name"), ["Ma"])
loader.add_jmes("url", "website.url", re=re.compile("http://(.+)"))
self.assertEqual(loader.get_output_value("url"), ["www.scrapy.org"])
def test_get_jmes(self):
loader = CustomItemLoader(selector=self.jmes_selector)
self.assertEqual(loader.get_jmes("description"), ["paragraph"])
self.assertEqual(loader.get_jmes("description", TakeFirst()), "paragraph")
self.assertEqual(loader.get_jmes("description", TakeFirst(), re="pa"), "pa")
self.assertEqual(
loader.get_jmes(["description", "name"]), ["paragraph", "marta"]
)
self.assertEqual(
loader.get_jmes(["website.url", "logo"]),
["http://www.scrapy.org", "/images/logo.png"],
)
def test_replace_jmes(self):
loader = CustomItemLoader(selector=self.jmes_selector)
self.assertTrue(loader.selector)
loader.add_jmes("name", "name")
self.assertEqual(loader.get_output_value("name"), ["Marta"])
loader.replace_jmes("name", "description")
self.assertEqual(loader.get_output_value("name"), ["Paragraph"])
loader.replace_jmes("name", ["description", "name"])
self.assertEqual(loader.get_output_value("name"), ["Paragraph", "Marta"])
loader.add_jmes("url", "website.url", re="http://(.+)")
self.assertEqual(loader.get_output_value("url"), ["www.scrapy.org"])
loader.replace_jmes("url", "logo")
self.assertEqual(loader.get_output_value("url"), ["/images/logo.png"])
def test_replace_jmes_multi_fields(self):
loader = CustomItemLoader(selector=self.jmes_selector)
loader.add_jmes(None, "name", TakeFirst(), lambda x: {"name": x})
self.assertEqual(loader.get_output_value("name"), ["Marta"])
loader.replace_jmes(None, "description", TakeFirst(), lambda x: {"name": x})
self.assertEqual(loader.get_output_value("name"), ["Paragraph"])
loader.add_jmes(None, "website.url", TakeFirst(), lambda x: {"url": x})
self.assertEqual(loader.get_output_value("url"), ["http://www.scrapy.org"])
loader.replace_jmes(None, "logo", TakeFirst(), lambda x: {"url": x})
self.assertEqual(loader.get_output_value("url"), ["/images/logo.png"])
def test_replace_jmes_re(self):
loader = CustomItemLoader(selector=self.jmes_selector)
self.assertTrue(loader.selector)
loader.add_jmes("url", "website.url")
self.assertEqual(loader.get_output_value("url"), ["http://www.scrapy.org"])
loader.replace_jmes("url", "website.url", re=r"http://www\.(.+)")
self.assertEqual(loader.get_output_value("url"), ["scrapy.org"])
def test_fluent_interface(self):
loader = ItemLoader(selector=self.selector)
item = (
loader.add_xpath("name", "//body/text()")
.replace_xpath("name", "//div/text()")
.add_css("description", "div::text")
.replace_css("description", "p::text")
.add_value("url", "http://example.com")
.replace_value("url", "http://foo")
.load_item()
)
self.assertEqual(
item,
{"name": ["marta"], "description": ["paragraph"], "url": ["http://foo"]},
)
itemloaders-1.3.1/tests/test_utils_misc.py 0000664 0000000 0000000 00000001557 14627331724 0020773 0 ustar 00root root 0000000 0000000 import unittest
from itemloaders.utils import arg_to_iter
class UtilsMiscTestCase(unittest.TestCase):
def test_arg_to_iter(self):
assert hasattr(arg_to_iter(None), "__iter__")
assert hasattr(arg_to_iter(100), "__iter__")
assert hasattr(arg_to_iter("lala"), "__iter__")
assert hasattr(arg_to_iter([1, 2, 3]), "__iter__")
assert hasattr(arg_to_iter(letter for letter in "abcd"), "__iter__")
self.assertEqual(list(arg_to_iter(None)), [])
self.assertEqual(list(arg_to_iter("lala")), ["lala"])
self.assertEqual(list(arg_to_iter(100)), [100])
self.assertEqual(list(arg_to_iter(letter for letter in "abc")), ["a", "b", "c"])
self.assertEqual(list(arg_to_iter([1, 2, 3])), [1, 2, 3])
self.assertEqual(list(arg_to_iter({"a": 1})), [{"a": 1}])
if __name__ == "__main__":
unittest.main()
itemloaders-1.3.1/tests/test_utils_python.py 0000664 0000000 0000000 00000004141 14627331724 0021351 0 ustar 00root root 0000000 0000000 import functools
import operator
import platform
import unittest
from typing import Any
from itemloaders.utils import get_func_args
class UtilsPythonTestCase(unittest.TestCase):
def test_get_func_args(self):
def f1(a, b, c):
pass
def f2(a, b=None, c=None):
pass
def f3(a, b=None, *, c=None):
pass
class A:
def __init__(self, a: Any, b: Any, c: Any):
pass
def method(self, a, b, c):
pass
class Callable:
def __call__(self, a, b, c):
pass
a = A(1, 2, 3)
cal = Callable()
partial_f1 = functools.partial(f1, None)
partial_f2 = functools.partial(f1, b=None)
partial_f3 = functools.partial(partial_f2, None)
self.assertEqual(get_func_args(f1), ["a", "b", "c"])
self.assertEqual(get_func_args(f2), ["a", "b", "c"])
self.assertEqual(get_func_args(f3), ["a", "b", "c"])
self.assertEqual(get_func_args(A), ["a", "b", "c"])
self.assertEqual(get_func_args(a.method), ["a", "b", "c"])
self.assertEqual(get_func_args(partial_f1), ["b", "c"])
self.assertEqual(get_func_args(partial_f2), ["a", "c"])
self.assertEqual(get_func_args(partial_f3), ["c"])
self.assertEqual(get_func_args(cal), ["a", "b", "c"])
self.assertEqual(get_func_args(object), [])
self.assertEqual(get_func_args(str.split, stripself=True), ["sep", "maxsplit"])
self.assertEqual(get_func_args(" ".join, stripself=True), ["iterable"])
if platform.python_implementation() == "CPython":
# This didn't work on older versions of CPython: https://github.com/python/cpython/issues/86951
self.assertIn(
get_func_args(operator.itemgetter(2), stripself=True),
[[], ["args", "kwargs"]],
)
elif platform.python_implementation() == "PyPy":
self.assertEqual(
get_func_args(operator.itemgetter(2), stripself=True), ["obj"]
)
if __name__ == "__main__":
unittest.main()
itemloaders-1.3.1/tox.ini 0000664 0000000 0000000 00000002015 14627331724 0015346 0 ustar 00root root 0000000 0000000 [tox]
envlist = py38,py39,py310,py311,py312
[testenv]
deps =
pytest
pytest-cov
commands =
py.test \
--cov-report=term --cov-report=html --cov-report= --cov=itemloaders \
--doctest-modules \
{posargs:itemloaders tests}
[testenv:extra-deps]
deps =
{[testenv]deps}
attrs
scrapy
[testenv:pypy3]
basepython = pypy3
[docs]
changedir = docs
deps =
-rdocs/requirements.txt
setenv =
READTHEDOCS_PROJECT=itemloaders
READTHEDOCS_VERSION=master
[testenv:docs]
basepython = python3
changedir = {[docs]changedir}
deps = {[docs]deps}
setenv = {[docs]setenv}
commands =
sphinx-build -W -b html . {envtmpdir}/html
[testenv:twinecheck]
basepython = python3
deps =
twine==5.0.0
build==1.2.1
commands =
python -m build --sdist
twine check dist/*
[testenv:typing]
basepython = python3
deps =
mypy==1.10.0
types-attrs==19.1.0
types-jmespath==1.0.2.20240106
commands =
mypy --strict --ignore-missing-imports --implicit-reexport {posargs:itemloaders tests}