/
usr
/
libexec
/
kcare
/
python
/
kcarectl
/
__pycache__
/
New Directory
Upload File
HOME
a �n�jWa � . @ s4 d Z ddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z mZmZmZ ddlmZ ddlmZmZmZ dZ dZ dZ d Z d Z d Z!d Z"d Z#dZ$dZ%dZ&dZ'G dd� de j(�Z)G dd� de*�Z+dd� Z,dd� Z-dd� Z.d d � Z/G d d � d e*�Z0d d!� Z1d"d#� Z2d�d%d&�Z3d'd(� Z4ej5d)d*� �Z6d+d,� Z7d-d.� Z8d/d0� Z9d1d2� Z:d3d4� Z;d5d6� Z<d7d8� Z=d9d:� Z>d;d<� Z?d=d>� Z@d?d@� ZAdAdB� ZBdCdD� ZCdEdF� ZDdGdH� ZEe0dIe4�e0dJe3dKgdLdM��e0dNe6�e0dOe7�e0dPe8�e0dQe3dQg��e0dRe9�e0dSe1dT��e0dUe1dV��e0dWe1dX��e0dYe1dZ��e0d[e1d\��e0d]e1d^��e0d_e1d`��e0dae1db��e0dce1dd��e0dee2dfdg��e0dhe:�e0die2djdk��e0dle;�e0dme<�e0dne=�gZFe,dodp�Ge#��e,dqdq�e,drds�Ge#��e,dtdu�e-dvdw�e,dxdy�e.dzd{�e.d|d}�e.d~d�e.d�d��e.d�d��e.d�d��e.d�d��e.d�d��e.d�d��e-d�d��e.d�d��e-d�e �e.d�d��e,d�d��e,d�d��e,d�d��e,d�d��e/d�e>�e,d�d��e-d�d��e/d�e@�e,d�d��e,d�d��e/d�eA�e-d�d��e,d�d��e/d�eB�e,d�d��Ge#��e-d�e �e-d�d��e/d�eC�e/d�eD�e,d�d��e,d�d��e,d�d��e/d�eE�e,d�d��e,d�d��g,ZHejIe jJd��d�d�� �ZKd�d�d��ZLdS )�aO Doctor v2 vitals collector (KPT-5984). Python port of the kcdoctor.sh (1.0-8) vitals collection: gathers the same data set into a DataPackage-based archive processed by the patch server generic uploads pipeline. Scalar and short values are consolidated into a single ``doctor.json`` (the eportal doctor precedent), one field each, instead of a tiny archive entry per value; bulk data (logs, command dumps, package lists, sysfs tunables, crash artifacts) stays as separate archive entries. The 1:1 inventory of collected items lives in docs/features/KPT-5984-doctor-v2/parity-inventory.md. � N) �auth�config�config_handlers� delivery_kit� http_utils�kcare� log_utils�platform_utils�serverid�utils)� run_command)� HTTPError�URLError�httplibz2.0-1z/boot/grub2/grub.cfgz/etc/kdump.confz/var/log/dpkg.logz/etc/apt/sources.list*z/etc/yum.repos.d/*z/var/log/libcare/*.logi' z/sys)zenable*znr_*zmax_*z*cnt*)z /sys/kernel/debug/tracing/eventsz/sys/kernel/tracing/eventsz /sys/devices)z [0-9]*.logzkmsg*logc @ s e Zd ZdZdZedd� �ZdS )�KcarectlDoctorPackagezkcarectl-doctorz/upload/kcarectl-doctor/c C s t jS �N)r � DOCTOR_REPORT_MAX_SIZE_BYTES)�self� r �,/usr/libexec/kcare/python/kcarectl/doctor.py�max_sizeF s z KcarectlDoctorPackage.max_sizeN)�__name__� __module__� __qualname__� data_type� upload_uri�propertyr r r r r r B s r c @ s e Zd ZdZdd� Zdd� ZdS )�CollectionSpecz�A single bulk vitals item ported from kcdoctor.sh. Collection failures are recorded in errors.log and never abort the report: an unhandled exception inside the package context manager would remove the whole archive. c C s || _ || _d S r )�arcname�_collect)r r �collectr r r �__init__W s zCollectionSpec.__init__c C sP z| � || j� W n8 tyJ } z |�d�| j|�� W Y d }~n d }~0 0 d S �Nzfailed to collect {0}: {1})r r � Exception� log_error�format)r � data_package�er r r �apply\ s zCollectionSpec.applyN�r r r �__doc__r! r( r r r r r O s r c s � fdd�}t | |�S )z4`run "<cmd>"` kcdoctor.sh helper (no shell features)c s | � |� � d S r )� add_stdout�r& r ��cmdr r r h s zrun_spec.<locals>.collect�r )r r. r r r- r �run_specd s r0 c s � fdd�}t | |�S )z4`dump "<path>"` kcdoctor.sh helper for regular filesc s | j |� d� d S )N��src_path)�add_filer, ��pathr r r s s zdump_spec.<locals>.collectr/ �r r5 r r r4 r � dump_speco s r7 c s � fdd�}t | |�S )z�`dump "<path>"` for /proc and /sys files. tar reports them as 0-size so they are first read into memory (see the same approach in anomaly.prepare_kernel_anomaly_report). c s` t j�� �s | �d�� �� d S t� d��"}| j||�� d� W d � n1 sR0 Y d S )N�file not found: {0}�rb�� data_bytes)�osr5 �existsr$ r% �openr3 �read)r& r �fr4 r r r � s z proc_dump_spec.<locals>.collectr/ r6 r r4 r �proc_dump_specz s rA c C s t | |�S r r/ )r �fnr r r � callable_spec� s rC c @ s e Zd ZdZdd� Zdd� ZdS )� FieldSpecz�A single doctor.json field ported from kcdoctor.sh. Collection failures are recorded in errors.log and never abort the report; a collector returning None omits the field (e.g. a distro-specific source file absent on this system). c C s || _ || _d S r )�keyr )r rE r r r r r! � s zFieldSpec.__init__c C s^ z | � � }W n: tyF } z"|�d�| j|�� W Y d }~d S d }~0 0 |d urZ||| j< d S r" )r r# r$ r% rE )r r& �report�valuer' r r r r( � s zFieldSpec.applyNr) r r r r rD � s rD c s � fdd�}|S )z�`dump "<path>"` as a doctor.json string field. Read in binary (non-utf8 safe) and decoded leniently; an absent distro-specific source file simply omits the field (None). c sR t j�� �sd S t� d��$} | �� �dd��� W d � S 1 sD0 Y d S )Nr9 �utf-8�replace)r< r5 r= r> r? �decode�strip)r@ r4 r r r � s zread_field.<locals>.collectr )r5 r r r4 r � read_field� s rL c s" t �t�|���� �fdd�}|S )z�`grep <pattern> <path>` as a doctor.json list-of-lines field. Match on bytes (like collect_apt_sources) so a non-utf8 byte cannot lose a line; the matched lines are decoded leniently for the JSON. c sX t j�� �sd S t� d�� } | �� �� }W d � n1 s<0 Y �fdd�|D �S )Nr9 c s" g | ]}� � |�r|�d d��qS )rH rI )�searchrJ ��.0�line)�regexr r � <listcomp>� � z/grep_field.<locals>.collect.<locals>.<listcomp>)r< r5 r= r> r? � splitlines)r@ �lines�r5 rQ r r r � s *zgrep_field.<locals>.collect)�re�compiler �bstr)r5 �patternr r rV r � grep_field� s r[ Fc s � �fdd�}|S )z@`run "<cmd>"` captured as a doctor.json field instead of a file.c s0 t � ddd�\} }} |pd�� }�r,|�� S |S )NT�� catch_stdout� catch_stderr� )r rK rT )�_�stdout�text��argv�as_linesr r r � s zcmd_field.<locals>.collectr )rd re r r rc r � cmd_field� s rf c C s t S r )�DOCTOR_VERSIONr r r r �collect_doctor_version� s rh c C s> z$t �t�d��} t�| �� ��� W S ty8 Y dS 0 dS )z�the public IP of the machine as seen by the patch server. Cached: the main_ip and server_id fields share one HTTP roundtrip per run. ZmyipZNAN)r �urlopenr �get_patch_server_url�nstrr? rK r# )�responser r r � get_main_ip� s rm c C s$ t �� } | st� �dd�} t�| �S )N�.r` )r � get_serveridrm rI r rk )� server_idr r r �collect_server_id� s rq c C s t �� S r )r �get_kernel_hashr r r r �collect_kernel_id s rs c C s> i } dD ]0}t dd| gddd�\}}}|p.d�� | |< q| S )N)�a�r�m�p�o�uname�-Tr\ r_ )r rK )�result�flagr` ra r r r � collect_uname s r} c C s\ t j�t�sdS ttd�� } | �� �dd�}W d � n1 s@0 Y dd� |�� D �S )z7`grep vmlinuz /boot/grub2/grub.cfg | sed 's/root=.*//'`Nr9 rH rI c S s" g | ]}d |v rt �dd|��qS )Zvmlinuzzroot=.*r_ )rW �subrN r r r rR rS z)collect_grub2_entries.<locals>.<listcomp>)r< r5 r= � GRUB2_CFGr> r? rJ rT )r@ rb r r r �collect_grub2_entries s .r� c C s t t�t��S r )�sorted�glob�YUM_REPOS_GLOBr r r r �collect_yum_repos s r� c C s dd� t jD �} | t �� d�S )z8presence-only port of the kcdoctor.sh detect_cp() probesc S s g | ]\}}|� r|�qS r r )rO �name�prober r r rR ) rS z)collect_control_panel.<locals>.<listcomp>)�cpZ softaculous)r �CONTROL_PANEL_PROBES�has_softaculous)Zpanelsr r r �collect_control_panel& s r� c C s t tj�p dS )z�the kcare.conf options kcarectl refuses to run on, named with the reason. --doctor is exempt from that refusal so a stuck host can still be diagnosed; the field is omitted (None) on a usable config. N)�dictr �INVALID_CONFIG_OPTIONSr r r r � collect_invalid_config_options- s r� c C s@ t ddgddd�\}}}| j|tj|p(d�dd�dd �d � d S ) z`ipcs -m | sed -e s/-/=/g`�ipcsz-mTr\ r_ rz �=rH ��encodingr: N)r r3 r rY rI )r& r r` ra r r r � collect_ipcs: s r� c C s| t j�t�s | �d�t�� dS ttd��&}dd� |�� �� D �}W d � n1 sV0 Y | j |d� |�d d� dS )z$`grep ' install ' /var/log/dpkg.log`r8 Nr9 c S s g | ]}d |v r|�qS )s install r rN r r r rR I rS z,collect_dpkg_install_log.<locals>.<listcomp>� r: ) r< r5 r= �DPKG_LOGr$ r% r> r? rT r3 �join)r& r r@ rU r r r �collect_dpkg_install_logA s 4r� c C s� t j�d�st j�d�r d}nZt j�d�rtd}zt| d� W qx typ } z | �d�|�� W Y d }~qxd }~0 0 nd}| �||� d S ) Nz /usr/bin/rpmz/bin/rpmzFrpm -q -a --queryformat="%{N}|%{V}-%{R}|%{arch}|%{INSTALLTIME:date}\n"z /usr/bin/dpkgzJ/usr/bin/dpkg-query -W -f "${binary:Package}|${Version}|${Architecture}\n"zdpkg.logz failed to collect dpkg.log: {0}z echo "unknown package manager")r< r5 r= r� r# r$ r% r+ )r& r � packages_cmdr' r r r �collect_packagesN s (r� c s\ t �d��g }tt�t��D ]N}tj�|�r`t�|�D ]&\�}}|� �fdd�t|�D �� q6q |� |� q |s�| � d� t�� dS g }|D ]�}z6t |d��}|�� }W d � n1 s�0 Y W nD t�y } z*| � d� || �� W Y d} ~ q�W Y d} ~ n d} ~ 0 0 tjd� |�d d �� |� � �fd d�|�� D �� q�| j|d �|�d d � dS )z/`grep -rE '^(deb|URIs)' /etc/apt/sources.list*`s ^(deb|URIs)c 3 s | ]}t j�� |�V qd S r )r< r5 r� )rO r� )�rootr r � <genexpr>j rS z&collect_apt_sources.<locals>.<genexpr>r8 Nr9 �failed to read {0}: {1}z{0}:rH r� c 3 s | ]}�� |�r� | V qd S r )rM rN )�prefixrQ r r r� rS r� r: )rW rX r� r� �APT_SOURCES_GLOBr< r5 �isdir�walk�extend�appendr$ r% r> r? r# r rY rT r3 r� ) r& r �filesr5 r` �namesrU r@ �contentr' r )r� rQ r� r �collect_apt_sourcesa s* *" r� c C s t t�t��}|s&| �d�t�� dS g }|D ]�}zDt|d��&}|�� �� t d� }W d � n1 sj0 Y W nB t y� } z*| �d�||�� W Y d}~q.W Y d}~n d}~0 0 t j d�|�dd�}|� |d� |� � q.| j|d� |�d d � dS ) z%`tail -n10000 /var/log/libcare/*.log`r8 Nr9 r� z ==> {0} <== rH r� r� r: )r� r� �LIBCARE_LOGS_GLOBr$ r% r> r? rT �LOG_TAIL_LINESr# r rY r� r� r3 ) r& r �paths�chunksr5 r@ rU r' �headerr r r �collect_libcare_logs� s 8"r� c s� t j�tj�sdS | �|d�tj�� t �tj�D ]`\}}}t|�D ]L� t � fdd�t D ��rDt j� |� �}t j� |tj�}| j d�|�|d� qDq2dS )z3crashreporter artifacts from /var/cache/kcare/dumpsN� ls -lR {0}c 3 s | ]}t � � |�V qd S r ��fnmatch�rO rZ �r� r r r� � rS z&collect_crash_dumps.<locals>.<genexpr>zcrashreporter/{0}r1 )r< r5 r� r � KDUMPS_DIRr+ r% r� r� �any�CRASH_DUMP_PATTERNSr� �relpathr3 )r& r r� r` r� r5 Zrel_pathr r� r �collect_crash_dumps� s r� c C s� t j�t�sd S t�� }t j�|�s(d S | �|d�|�� t t � t j� |d���D ]8}t j�|�sdqR| j d�t j� |��t j� |d�d� qRd S )Nr� �*zkdump/{0}/vmcore-dmesg.txtzvmcore-dmesg.txtr1 )r< r5 r= � KDUMP_CONFr �get_kdump_rootr� r+ r% r� r� r� r3 �basename)r& r � kdump_rootZevtr r r � collect_kdump� s �r� c s� g }t �t�D ]�\}}}|�t�r0g |dd�< qt|�D ]�� t� fdd�tD ��r8t j� |� �}z6t |d��}|� � }W d � n1 s�0 Y W n t y� d}Y n0 |� t�d�|��| d � q8q| j|d� |�d� dS ) zGtunables gathered by the kcdoctor.sh get_sysfs_info() `find /sys` callsNc 3 s | ]}t � � |�V qd S r r� r� r� r r r� � rS z'collect_sysfs_config.<locals>.<genexpr>r9 rS zcat {0} r� r: )r< r� � SYSFS_ROOT� startswith�SYSFS_EXCLUDE_PREFIXESr� r� �SYSFS_NAME_PATTERNSr5 r� r> r? r# r� r rY r% r3 ) r& r r� r� �dirsr� r5 r@ r� r r� r �collect_sysfs_config� s * "r� Zdoctor_versionZ virt_whatz /usr/libexec/kcare/virt-whatT)re Zmain_iprp Z kernel_id�datery Zredhat_releasez/etc/redhat-releaseZdebian_versionz/etc/debian_versionZ os_releasez/etc/os-releaseZissuez /etc/issueZsysconfig_kernelz/etc/sysconfig/kernelZ proc_uptimez /proc/uptimeZ proc_loadavgz /proc/loadavgZ proc_cmdlinez /proc/cmdline� proc_versionz /proc/versionZ default_grubz/etc/default/grub�DEFAULTZ grub2_entriesZ sshd_portz/etc/ssh/sshd_configZPortZ yum_repos_dZ control_panelZinvalid_config_options�syslogztail -n{0} /var/log/syslog�dmesg�messagesz tail -n{0} /var/log/messages�ls_var_cache_kcarezls -lR /var/cache/kcare/z kcare.confz /etc/sysconfig/kcare/kcare.conf�cpuinfozcat /proc/cpuinfoZ proc_vmstatz /proc/vmstatZ proc_devicesz /proc/devicesZproc_diskstatsz/proc/diskstatsZ proc_mdstatz /proc/mdstatZ proc_meminfoz /proc/meminfoZ proc_swapsz /proc/swapsZproc_filesystemsz/proc/filesystemsZ proc_mountsz /proc/mountsZproc_interruptsz/proc/interruptsz grub.confz/boot/grub/grub.conf� proc_modulesz /proc/modulesz grub2.cfgZ proc_zoneinfoz/proc/zoneinfoZls_boot_configsz:ls /etc/grub.conf /boot/grub/grub.conf /boot/grub/menu.lstZls_bootz ls -l /bootZprintenvZ dmidecoder� �sysctlz sysctl -az sysctl.confz/etc/sysctl.confz packages.listZlspciz lspci -vvZdpkg_lzdpkg -lZ apt_sourceszyum.confz /etc/yum.confZ yum_repolistz yum repolistZ libcare_logsz kcarectl.logz tail -n{0} /var/log/kcarectl.logz kdump.confz kcare-cronz/etc/cron.d/kcare-cronzcrashreporter/lszkdump/lsZ aa_statusz aa-statusZsestatusZ kprobes_listz"cat /sys/kernel/debug/kprobes/listZ sysfs_configZlsblkzlsblk -fZdfzdf -h)�loggerc C sh t � } | �J i }tD ]}|�| |� q| �d|� tD ]}|�| � q6W d � n1 sZ0 Y | S )Nz doctor.json)r � FIELD_SPECSr( �add_json� FILE_SPECS)r& rF �field�specr r r �prepare_doctor_report/ s *r� c C s |s t jr4|rdnd}tjd�|�dd� | � dS t�� du rXtjddd� | � dS t� }|du rzt�d� | � dS z�z0|� � }tjd �|�dd� t � d �|�� W nF t t tjtjfy� } z"t�d �|�� | � W Y d}~n d}~0 0 W |�� n |�� 0 dS ) a� Collect and upload the doctor report via the generic uploads proxy. PUTs the archive to ``/upload/kcarectl-doctor/`` -- the same modern pipeline as ``kernel-anomaly`` -- which ePortal proxies transparently to the patch server, so no ePortal change is required. Any 2xx reply is success, including the ``REDUCED_REPORT`` ``204`` where the proxy intentionally drops the report (``send()`` returns without raising, so we do not fall back). Falls back to the legacy ``kcdoctor.sh`` flow (``fallback``) when: * ``force_fallback`` (the ``--doctor --fallback`` CLI flag) or ``config.FORCE_DOCTOR_FALLBACK`` is set -- the rollout kill-switch. ``--fallback`` is the one-off, per-invocation form; the config knob is settable per host in kcare.conf or pushed fleet-wide via the ``KC-Flag-Force-Doctor-Fallback`` feature flag. Either forces the legacy flow on every host, registered or not, before the new path is even considered; * the host is not registered, i.e. there is no auth string (OQ-4: the proxy upload requires auth, so the new path is not even attempted); * report collection failed (``prepare_doctor_report`` returned None); * the new route is unavailable on an older patch server / ePortal -- a 404 / 405 / 403, surfaced as ``HTTPError``, or a connection-level failure. ``send()`` drives ``http_utils.upload_file`` (raw ``httplib``, not ``urllib``), so transport failures surface as an ``OSError`` subclass -- ``socket.timeout``, ``ConnectionRefused``, ``socket.gaierror`` (DNS), ``ssl.SSLError`` -- or an ``httplib.HTTPException``, *not* ``URLError``; all of these mean the new path is unusable. ``upload_file`` raises only on status >= 400, so a redirect is not treated as a failure; the proxy and patch server never redirect uploads. :param fallback: legacy ``kcdoctor.sh`` runner, injected by the caller (KPT-6065) to keep this module free of an ``__init__`` import cycle. :param force_fallback: when True (``--doctor --fallback``), skip the v2 path and run ``fallback`` directly; the per-invocation form of ``config.FORCE_DOCTOR_FALLBACK``. z--fallback requestedz FORCE_DOCTOR_FALLBACK is setz.doctor: {0}; using the legacy kcdoctor.sh flowF)� print_msgNzAdoctor: host is not registered; using the legacy kcdoctor.sh flowzCdoctor: report collection failed; using the legacy kcdoctor.sh flowz doctor: report uploaded as {0}z,Please provide this filename to support: {0}zJdoctor: report upload unavailable ({0}); using the legacy kcdoctor.sh flow)r �FORCE_DOCTOR_FALLBACKr �loginfor% r �get_http_auth_stringr� �logwarn�sendr � print_wrapperr r r � HTTPException�socket�error�remove_archive)�fallback�force_fallback�srcr& � upload_name�errr r r �send_doctor_reportD s8 ) � � r� )F)F)Mr* r� r� r< rW r� �kcarectlr r r r r r r r r r Zkcarectl.process_utilsr Z kcarectl.py23r r r �Any�Callable�Dict�List�Optionalrg r r� r� r� r� r� r� r� r� r� r� � DataPackager �objectr r0 r7 rA rC rD rL r[ rf rh �cachedrm rq rs r} r� r� r� r� r� r� r� r� r� r� r� r� r� r% r� � catch_errorsr� r� r� r r r r �<module> s� 0 # � �0